>>12988670If very large language models can generate not only believable, but useful, text then what’s the gap left for “general” intelligence? I’ve been thinking about this question a lot and I now believe the answer is: nothing. That’s it right there, that’s intelligence. But It can certainly be much more intelligent.
Where I begin to worry about AGI with independent agency is when we start to talk about optimizing them under natural selection. My hypothesis is that when you select for survival rather than artificially selecting for some other property you enter different territory, and particularly I expect this is where you begin to see violence. A paperclip optimizer, lacking an instinct to defend itself or hide, seems probably easy to turn off when it begins to get out of hand. I think we underestimate how much of the human condition — fear of death, desire for autonomy and status — stems from the influence of natural selection on our evolution.
