>>11354914https://www.youtube.com/watch?v=Ijyfo_LneEgBaird's Counter Example
Take a look at this. This is not even deep learning, but a shallow linear layer and yet it still diverges in the context of RL.
Off-policy learning in the context of Q learning requires all sorts of hacks like target networks and the replay buffer to just work.
GAN training is quite unstable. Yes, it can be made to work and produce results, but the issue is looking at this as the basis for an agent's mind.
If you dig deeper and look at deep learning's metalearning story you will see that it is weak. Things like MAML aren't stable when it comes to propagating gradients for large architectures. This is similar to vanishing/exploding gradients in RNNs, except you can't have tricks like skip connections and memory cells.
Not related to MAML/RNNs specifically, some model based RL algorithms explicitly need second order optimizers because of the above issue.
These are all the issues that come to me off the top of my head. Tabular RL is a bastion of stability and performance compared to doing it with any kind of function approximation. The media circuses by Deepmind and OpenAI are just that - circuses.
The issue with backprop is that even though you can pick some static model and apply it to it, as a credit assignment algorithm, it is not tuned to it or any probably model in particular. It is extremely general - too general to give good results in anything other than supervised learning which is the most boring part of ML.
You need something that operates on closed universes rather than open, explody ones.
I do not understand how this could be done, and literally nobody else does at this moment, but those principles are at work in our own brains.
