Unless you're talking about several generations apart, then you'd somewhat have a point.
First cousins is still incest.
And in that matter, let's take a first-cousin mating as example. First cousins share a set of grandparents. For any particular gene in the male, the chance that his female first cousin inherited the same gene from the same source is 1/8. Further, for any gene the man passes to his child, the chance is 1/8 that the woman has the same gene and ½ that she transmits that gene to the child so 1/8 X ½ = 1/16. Thus, a first-cousin marriage has a coefficient of inbreeding F =1/16.
Also to say that inbreeding is safe if no genetic problems are present is ridiculous and completely ignores factual evidence. Genetic problems aren't like a disease, you can't just not have them. Just because a recessive gene hasn't been expressed in a population doesn't mean it's not there, and it doesn't mean it's completely safe to marry your cousin.