>>12735558>>humans will go extinct if ai gitz gud>>idk because they'll be better than us at maths or something. That's why Neanderthals are aren't around anymore init.This isn't an extreme point of view.
Fiirstly, lots of attempts at getting a proto-AI to do something we explicitly want result in lots of things we don't want; for example, see utility monsters.
A utility monster is a """thought experiment""". Say we try to implement a utilitarian approach, to maximize happiness, through an AI. What happens if there exist anomalous humans? For example, if using all our resources on a small number of humans will produce the maximum happiness, even though the vast majority will suffer, then the AI will do that; if we give it a simple "maximize happiness" goal.
The problem concerns what goals we should give an AI, and how to actually implement that as code, then checking the goals actually produces what we want.
Secondly, a simple example of utility monsters and AI are societies and corporations. A corporation will layoff vast numbers of employees if it improves its monetary goals for those who aren't laid off, even though (overall) the layoffs are bad on average; for those the laid off and those who remain.
Similarly, by having a hypercapitalist society, we end up with things we really dont want. For most businesses, if they tried to be humane it would decrease their profit margin. Pay a reasonable wage for expendable workers? have adequate healthcare for expendable workers? Pay taxes? If its profitable to avoid this, then they will avoid it.
The knee jerk solution is governmental regulation to make it unprofitable to be inhumane. The counter adaptation is large companies spending less money to lobby policy, than they would pay in taxes (and etc) under those policies.
By simply saying "maximize profits" we get behavior we dont want.
AIs and societies and companies dont develop solely based on good intentions
