>>11291731>tons of research is obsoleted.Rather the opposite, it's just the same shit but scaled up. Beside VAEs, dropout, LSTMs/GRUs (same shit), ADAM (or any other adaptive learning algorithm like adadelta, in the end it's all the same shit - it makes the lr a tons less sensitive, saving hundreds of hours of work, but doesn't otherwise lead to better results and usually the opposite) and GANs (maybe unsupervised pretraining but nobody's even been using that in forever either thanks to computational improvements), there hasn't been anything new all this time (OK, maybe neural ODEs, but they're not yielding anything yet - maybe they'll become super cool like GANs being useless originally but being the norm now). Most advances are just "we ran the same model but bigger and the results were nicer!" garbage.
The other axis of learning, reinforcement learning, is mostly the same too. It's just hillclimbing or dynamic programming, but you use non-deterministic exploration and simultaneously estimate the cost of moves and the best solution instead of assuming the cost of move (as in A*/dijkstra, and other such methods).
Sutton is right in this blogpost:
http://incompleteideas.net/IncIdeas/BitterLesson.htmlThe other way to look at it is that because these methods are general and scale, they shall endure; whereas methods that rely too much on domain knowledge falter as the knowledge becomes invalidated or as more complex knowledge is uncovered, because general methods can exploit data to automatically detect this knowledge without human effort. You just replace 40 years of research into 40 hours of pseudo-magic mathematical optimization.
>>11291754Big tech (like google) is really not leading in terms of research, it's really just "we ran an academic's project on big data and it worked please give award" bullshit. They almost never provide anything new. The latest was DRAW by deepmind and that's literally it.
Most papers in any field is not worth reading, period.
