A program which recursively improves its operation to better satisfy some set of goal metrics must by definition be part of the complexity class RE.
With this in mind, it seems that AI safety is a hopeless endeavor. As to know whether the AI converges on the constraints defined in the goal set or if it recursively increases its own operational resources forever is a halting-equivalent problem.

So how fucked are we? I guess our only hope is looking for a turing oracle somewhere in the laws of physics.