Perhaps the slaves are people they picked up at the colony. They were mostly workers.