Economic disparity is a bit reductive, but it's easily the biggest factor without any doubt.
Connected to it there's the well known phenomenon in sociology that "whatever is believed to be real will become so in its consequences." To make an example, imagine meeting someone for the first time, they do something just slightly aggressive for whatever reason. You then start thinking they hate you, and so you start behaving in a hostile manner, which brings them to act accordingly. To cut a long story short, the belief you had became reality through the behavior you adopted to face it.
If, as a nation, you firmly believe that certain races or social classes are dumber, you will take your children out of those schools where they are an important population, fund those schools less and hire worse teachers, because you believe more resources dedicated to those schools would be wasted.
And just like that you fufilled your own prophecy.