I'm going to call them things like "commoners" and "workers" here. I get that they're not exlusively used for that purpose, but I'm factoring in that it would have been normal for them to fill those type of roles.
If they're going to be a basic race of, effectively, workers and commoners, there's a lot of room for genetic diversity, anyway. Worker races or, in the case of real Earth history, worker classes, almost always outbreed the 'higher' class citizens. The Jews in Egypt, the peasants of medieval Europe, even the lower class citizens of Western civilization today. They all outbreed those in the higher tiers of society. So, a wide range of shapes and even colors wouldn't be unusual without them being nanomachines.
As far as the numbers go, if it got to the point where they weren't needed any more? Then yeah, maybe they were more common. They could have recolonized somewhere else when they weren't needed or gone underground for the bulk of their society, or something to that effect.