From a legal standpoint, when testing whether an institution or law is racist, it’s not sufficient to just examine it for explicit racism; it’s important to consider the results for a disparate impact on racial groups.
For example, many of the Jim Crow laws in the south were written to exclude poor and uneducated people, using means that were not explicitly racist such as poll tests yet that resulted in a disparate impact of African Americans being barred from voting. It’s also well understood historically that such laws were written with the intent of preventing African Americans from voting.
Now, the modern mathematics establishment may not be intentionally excluding a disproportionate number of blacks. However, when we examine the results for disparate impact the data is clear: mathematics is biased against black people such that they consistently are less represented than whites at every level from passing algebra classes to research and tenured professorships.
It’s not immediately obvious how exactly the institution of mathematics produces such a racial bias. But we can’t deny the results. As faculty, teaching assistants and members of the institution of mathematics it is the duty of each one of us to work to overcome this institutional bias and ensure fair and equal outcomes for all students, regardless of race.
In the meantime, yes, it’s probably more fair that we do not subject black students to a clearly biased selection process in which they are less likely to succeed than their white classmates such as required math classes for an unrelated degree that’s needed for a non technical job.