There's the fact that they have Boulder too though. Any mention of 150k or so people that are completely safe and well cared for, I'm betting the Colony residents would turn on the Mallers at the drop of a hat. Especially if the military guys didn't force them into coming with them. All they'd have to say is "stay here and fight us, or go to Boulder where it's safe and we'll take you there." Take it or leave it, I'm betting most would take that in a heartbeat.