>>12870511>>12871072He's actually correct, in a sense, although 'fitness' is not the proper term.
I was just reading a paper on this recently, actually. It has to do with group selection and dual inheritance theory.
Individually, you would indeed have to work more in a post-agrarian, sedentary community, but individuals would not generally be consciously aware of this, and what ultimately matters is that these groups could grow bigger and faster than hunter-gatherers.