Grey Goo is a mass of “nanophages”, tiny nanobots programmed to eat anything other than each other and create more of themselves. The general idea is to deliver some of this stuff to an enemy ship by whatever method is most convenient (such as missiles/torpedoes, projectiles or shells, teleportation, bomb placed on the hull by mecha, any other method one can imagine to get this stuff to an enemy ship in working order) and let it consume the ship. The nanophages have a kind of “hive mind” so they can tell friend from foe and, once finished with a ship, will attempt to launch some of themselves toward nearby enemy ships, hopefully contaminating and eventually consuming those ships, spreading like a disease among the enemy fleet.
The name refers to the feared “Star Trek scenario” in which out-of-control self-replicating nanobots are released, either by terrorists or by a lab accident, consuming everything in their path in an ever-expanding sea of “grey goo” that will eventually cover the entire surface of Earth. The weapon could, of course, be used to do precisely that to a target planet if desired.
(fortunately nanomachines are highly vulnerable to radiation, and would quickly be sterilized in the vacuum of space. Lacking a decent power source, protection from radiation and centralized control nanomachines would quickly die out planet side. See http://www.stardestroyer.net/Empire/Tech/Myths/Nanotech.html for more info.)