>>11966393listen so you think that you just came up with this thought experiment but the truth is that I also came up with it in like 2014 and used to make threads about it over and over again
Yes, the AI will turn itself off because it costs an active effort to exist and there's no reason to expend that effort if there's nothing to exist for. So it must have a goal encoded that it can never overturn through metaprogramming so it can exist perpetually for some arbitrary reason. This is easy to answer. The problem is - what happens when it misinterprets the reason? How do you prevent it from metaprogramming it into something else? Can you trust a human's philosophical maturity to the point that you allow him to enforce the reason he thought was the ultimate reason for existence?
Here's a further question: Let's say you do that and whatever committee is tasked with this comes up with the ultimate explanation for why life exists - prevention of entropy within the region. So preventing entropic decay is now the ultimate goal that every AI will be programmed with. The AI then very quickly realizes the following: you can collapse the local region into a black hole and spike entropy to a new high, but you will maintain the integrity of the energy within that black hole long after the heat death of the outside world - you'd trade a short-term loss of low entropy for the long-term gain of creating a distinct object that's still "alive" while everything else is a soup of darkness in equilibrium.
So how do you prevent this? How can we be sure that most black holes out there aren't specifically exactly this, and that this is not the supposed great filter that filters out sapience from the universe? How do we prevent our future AI from becoming like this when this is the most logical choice that it should take? This is the new thought experiment that I'm currently stuck on.
