In Greek Folklore the Dorian tribe of ancient Greece trace their ancestry back to the union of Herakles and the 50 daughters of King Thespius whose sons were chased from Greece only to return centuries later as a massive tribe who would conquer most of Greece.

While not a exact example the story lead me to question, assuming there was no further outside genetic input, like on a colony on a far away planet, how many unrelated women would it take to create a genetically healthy population from one unrelated man?