While other unchangeable aspects of your life have a limited, mostly ignorable effect on your life - such as not having a singing voice, or being genetically very limited in your athletic abilities' ceiling - intelligence in the sense that we've understood it intuitively for the longest period seems to be the best human-controllable predictor (so not taking luck into account) for one's ability to maneuver himself in the world, to compete with other people in mentally demanding tasks, to be creative or original, to develop and perform highly sought after skills or to innovate; a good predictor for material success, and adaptability, at least within the western world. Stupid people are always being held down, outsmarted, outwitted, used and oppressed, because that is the way of the world and the way our hierarchies have worked for the longest time.
So it is not a lack of self-awareness that makes me concerned that I could be less intelligent than I believed. It is the belief that, if I were not to be intelligent, my entire future would be mostly certain and limited to a life I would not want to live, that makes me fear for this outcome. I "can't be unintelligent", because then I might as well just not be - a life enduring inevitable failure after failure is not a life worth living. This is an intentional, fully accounted-for lapse of rationality, it's called hope; it's basically a coping mechanism.