/sci/ - Science & Math » Thread #11298923

9KiB, 221x250, brailet wearing brain.jpg

View Same Google iqdb SauceNAO

Anonymous Mon 13 Jan 17:44:38 2020 No.11298923 View Reply Original Report

Quoted By: >>11298968 >>11299173 >>11299228 >>11299233 >>11299492 >>11299750 >>11299762 >>11299763 >>11299924 >>11299946 >>11299974 >>11300036 >>11302192 >>11302253

Is it possible to correctly forecast stock prices with deep learning?

Anonymous

Anonymous Mon 13 Jan 2020 18:06:49 No.11298968 Report

Quoted By: >>11299027 >>11300036

>>11298923
As far as we know, no. There are publicly available stock price datasets you can have fun with but the predictions never work in practice for some reason. By work I mean "to a sufficient degree" (you can easily do better than random but that's about it).

Anonymous

Anonymous Mon 13 Jan 2020 18:24:49 No.11299027 Report

Quoted By: >>11299050

>>11298968
Because every new datapoint can and probably does depend on unique variables, thus making predictability hard.

Anonymous

Anonymous Mon 13 Jan 2020 18:33:58 No.11299050 Report

Quoted By: >>11299161

>>11299027
I'd say it's rather because the course of the market is mostly decided by news, rumors, mood of whales, etc. That's why a lot of modern prediction work (especially those using DL) focus on leveraging realtime news APIs and emotion prediction in conjunction with company stock history to predict the way it's going.
Another issue with DL is that there aren't any good methods to leverage long-term dependencies, although things like hierarchical or stacked LSTMs don't do too poorly on text corpora.
That said, DL still seems to be competitive or even SOTA in these kinds of tasks, it's just that market prediction is either hard or impossible (due to too much hidden information). Perhaps an RL-MCTS approach like used by poker bots (which are designed to account for hidden information, as poker is a great example of a POMDP with sequential information streaming, time constraints and little O's in the PO part) can help there.

Anonymous

Anonymous Mon 13 Jan 2020 19:36:49 No.11299161 Report

Quoted By: >>11299183 >>11299230

bumping thread with cutie girl
https://www.youtube.com/watch?v=JntA9XaTebs
>>11299050
>sentiment analysis and NLP
thats what i heard several times also. What i should scrape for news and moods? Bitcoin Forum? What else? Also what is "dictionary" i need to compare posts with? Like how to classify posts from negative to positive? Simplest algorithm i can think about:
1.) scrape posts from bitcoin forum/twitter
2.) use regex to find if post is about certain asset (for example look for #bitcoin in text of post)
3.) classify if post is negative or positive opinion
What else? Is this method better than time series forecasting? How long such classification can take when it comes to computation time of algorithm? Do i need gpu for NLM?

Anonymous

Anonymous Mon 13 Jan 2020 19:44:33 No.11299173 Report

Quoted By: >>11299177

>>11298923
No. There's model that can predict the economy and make money. Do you know why? The economy runs on insider information shared between a closed network of families globally with some straps thrown to those who don't know their place and nothing to the rest. All your theorising is worthless. All that matters is a bit of info or a favour from someone who has power.

Anonymous

Anonymous Mon 13 Jan 2020 19:45:36 No.11299177 Report

Quoted By:

>>11299173
there's no model*

Anonymous

Anonymous Mon 13 Jan 2020 19:47:37 No.11299183 Report

Quoted By: >>11299208 >>11299230

>>11299161
Don't tell me you're really thinking of using some sort of 'dictionary' to perform sentiment analysis. Sentiment analysis is done with machine learning. You clearly don't know what you're talking about

Anonymous

Anonymous Mon 13 Jan 2020 19:48:49 No.11299185 Report

Quoted By: >>11299778 >>11299798

Stock trader here, the answer is no for deep learning models (I have tried everything in my attempts to day trade using robots), but if you understand how the market works you can develop a quant strategy that will work more often than not.

Hint: if it has more than 2 or 3 free parameters it's near certainly overfit. Keep it simple: don't trade futures (zero sum game), understand momentum, mean reversion and volume profile, and don't leverage otherwise you will inevitably go broke

Anonymous

Anonymous Mon 13 Jan 2020 20:00:01 No.11299208 Report

Quoted By:

>>11299183
dictionary i mean: "bullish", "bearish" stuff like that, you need to classify posts to positive/negative in some way. Do you know how to do it?

Anonymous

View Same Google iqdb SauceNAO Bayes' Theorem.jpg, 167KiB, 1200x769

Anonymous Mon 13 Jan 2020 20:07:34 No.11299228 Report

Quoted By:

>>11298923
Yes

http://www.bayesianinvestor.com/blog/

Anonymous

Anonymous Mon 13 Jan 2020 20:08:54 No.11299230 Report

Quoted By: >>11301873

>>11299161
A GPU will always net you massive performance gains that are not worth losing on, but you don't need anything particularly big. A single GTX 1080Ti will do.
You shouldn't use a simple dictionary-based method for sentiment analysis because you become weak against sarcasm and other non-straightforward language. Moreover you don't know what the positive words refer to and you can't do comparison such as "I think GOGL is going to overtake MSFT". Sentiment analysis SOTA uses DL as >>11299183 says, look up the literature. Almost everything in machine learning is open-access and you can find SOTA results by simply browsing arxiv (use arxiv sanity preserver for an even easier time locating interesting things).
Generally, you want to use large open-access datasets to learn the original sentiment analysis model and then finetune it on specific economy-related corpora (you could for example take a second generic dataset and filter it by the presence of terms that are relevant to finance/economy, like bull/bear, names of stocks, the word 'stock', etc.; but don't do that for the sentiment analysis itself, it's nowhere near robust).

Anonymous

Anonymous Mon 13 Jan 2020 20:09:54 No.11299233 Report

Quoted By:

>>11298923
Deep learning is a meme

Anonymous

Anonymous Mon 13 Jan 2020 21:32:41 No.11299453 Report

Quoted By:

not possible because stocks are influenced by humans and humans are unpredictable

Anonymous

Anonymous Mon 13 Jan 2020 21:42:21 No.11299492 Report

Quoted By: >>11301462

>>11298923
you shoudl ask this on /biz/ they have higher median iq than /sci/ anyways

Anonymous

Anonymous Mon 13 Jan 2020 23:04:22 No.11299750 Report

Quoted By:

>>11298923
No.

Anonymous

Anonymous Mon 13 Jan 2020 23:07:09 No.11299762 Report

Quoted By: >>11299974

>>11298923
No. The market is not ergodic.

Anonymous

Anonymous Mon 13 Jan 2020 23:07:12 No.11299763 Report

Quoted By:

>>11298923
>deep learning

Anonymous

Anonymous Mon 13 Jan 2020 23:12:02 No.11299778 Report

Quoted By: >>11299821 >>11299861 >>11299934

>>11299185
i tried LSTM model with 4 neurons, 500 epochs, for predicting next 12 days closing prices from dataset of 720 previous closing prices (january 2018-january 2020). After 10 retrains (which took about 5 hours using google colab) this is best fit i got. I am not very satisfied with results

Day 0 PREDICTED: 1.8411615309112137 EXPECTED: 1.79745307
Day 1 PREDICTED: 1.8080489415817558 EXPECTED: 1.86633305
Day 2 PREDICTED: 1.8503643820883973 EXPECTED: 1.88660977
Day 3 PREDICTED: 1.8965617278211833 EXPECTED: 1.88374105
Day 4 PREDICTED: 1.8700652226146575 EXPECTED: 1.90216989
Day 5 PREDICTED: 1.9108030186822107 EXPECTED: 1.8338084
Day 6 PREDICTED: 1.8254101089465808 EXPECTED: 1.7728187
Day 7 PREDICTED: 1.7805886367916595 EXPECTED: 1.81101867
Day 8 PREDICTED: 1.7993274258529752 EXPECTED: 1.74404587
Day 9 PREDICTED: 1.7534391402921443 EXPECTED: 1.80740436
Day 10 PREDICTED: 1.7929460803744186 EXPECTED: 1.83057064
Day 11 PREDICTED: 1.8390478294158392 EXPECTED: 1.80101951

Can someone give me advice, how can i improve this model?

Anonymous

Anonymous Mon 13 Jan 2020 23:16:19 No.11299798 Report

Quoted By: >>11299816

>>11299185
i am interested in algo trading, what is this "mean reversion" thing, and can i find it automatically based on historical prices, and base automatic trading strategy on it?

Anonymous

Anonymous Mon 13 Jan 2020 23:20:09 No.11299816 Report

Quoted By:

>>11299798
Not that guy, but those are basic concepts in time series and forecasting. If you google it, you'll literally have your answers within the first results.

Anonymous

Anonymous Mon 13 Jan 2020 23:30:50 No.11299821 Report

Quoted By: >>11299829 >>11299874 >>11299893 >>11300051

>>11299778
>4 neurons
lel. Start at 32, if it underfits put it up to 256. Also use multiple layers. What's your loss function? Use something like R2 to judge the actual quality of the fit instead of just using predicted/expected and do so on a suitably large test set.
Explain your data (resolution, sample quantity, how you split in train/test/valid sets, preprocessing if any, data format, etc.)

Anonymous

Anonymous Mon 13 Jan 2020 23:32:03 No.11299829 Report

Quoted By:

>>11299821
Also what the fuck is "10 retrains" supposed to mean, what alg are you using for optimization, and why are you waiting 500 epochs since it's clearly not improving after much fewer than 500 epochs?

Anonymous

Anonymous Mon 13 Jan 2020 23:42:02 No.11299861 Report

Quoted By: >>11299893

>>11299778
This is an overfit, do cross validation and you will see it

Anonymous

View Same Google iqdb SauceNAO carbon(4).png, 6KiB, 985x1006

Anonymous Mon 13 Jan 2020 23:46:56 No.11299874 Report

Quoted By: >>11299883

>>11299821
if you give me some mail, i can give you whole program, 10 retrains means repeats=10 in pic related i am looking for best fit (smallest RMSE of 12 predicted values)

Anonymous

View Same Google iqdb SauceNAO carbon (13).png, 171KiB, 985x817

Anonymous Mon 13 Jan 2020 23:49:44 No.11299883 Report

Quoted By: >>11299918

>>11299874
oops, also i am training data in scaled, differenced form to get stationarity

Anonymous

Anonymous Mon 13 Jan 2020 23:52:31 No.11299893 Report

Quoted By: >>11299926

>>11299821
this guy says its underfit
>>11299861
this guy says its overfit
I think i heard few times that less neurons/simpler net - better

Anonymous

Anonymous Tue 14 Jan 2020 00:03:08 No.11299918 Report

Quoted By: >>11300051

>>11299883
Increase the batch size as much as practical under 1024.
Your LSTM is wrong, you're giving a single timestep to your model for training, with however many timesteps you truly have (12 days I think you said above) as distinct features, whereas you want the transpose of that.
That's also why you have a dense(1) at the end (you should instead be extracting the last timestep).
Since you already have the machinery for that you should use early stopping to not wait out the entire 500 epochs. Don't bother with repeats at all, it's irrelevant.
I don't know what you're doing with your scaling, but if you don't apply the same scale on train and test you won't get good results. Verifying your predictions on the train set in light of the training loss will help see if your prediction code is working correctly.
You should also probably remove stateful and remove reset_state since you're not considering batches to be the same example (which is the main use of stateful) so I don't think you gain anything from this.

Anonymous

Anonymous Tue 14 Jan 2020 00:05:35 No.11299924 Report

Quoted By: >>11299928

>>11298923
First of all, deep learning is a meme technology that already hit a ceiling. Neural nets are slow as fuck too, that's another reason to ditch them completely

To make it in the stock market you just need to be faster than the others, that's it.

Anonymous

Anonymous Tue 14 Jan 2020 00:06:23 No.11299926 Report

Quoted By: >>11300051

>>11299893
You should always start by making sure it's overfitting because it means you're training. Training means you don't have a broken gradient and that your data isn't all-0, for example.
After it's overfitting (100% whatever-metric on training set), you should reduce capacity until you find that valid and train performance is good. Finally, you should test the resulting model on a 3rd and final set you haven't overfit your hyperparameters (e.g. regularization, layer sizes) on.
That guy who says it overfits doesn't really know. At just 4 neurons it's extremely unlikely, and the prediction you're showing can be a result of either overfitting or underfitting (you can really only tell by comparing the performance on valid/test and train in that regime, but if it was way off like predicted = 5.0 or whatever, it would be definite underfit).

Anonymous

Anonymous Tue 14 Jan 2020 00:07:24 No.11299928 Report

Quoted By: >>11299936

>>11299924
>separating dl and nn
You have proven you know nothing about the topic. Off you go, reddit.

Anonymous

Anonymous Tue 14 Jan 2020 00:09:00 No.11299934 Report

Quoted By: >>11299957 >>11299984 >>11300051

>>11299778
As the other garbage here can't tell you, me as good garbage is telling that you have less error if you make expectations two day forward instead of one.

Just get prices from day one predicted to day two and you are much more correct.

What data have you used?

Anonymous

Anonymous Tue 14 Jan 2020 00:09:25 No.11299936 Report

Quoted By: >>11299940

>>11299928
I never separated them because both things mean the same, learn how to read full sentences.

Anonymous

Anonymous Tue 14 Jan 2020 00:09:50 No.11299940 Report

Quoted By: >>11299947

>>11299936
OK reddit.

Anonymous

Anonymous Tue 14 Jan 2020 00:10:57 No.11299946 Report

Quoted By:

>>11298923
It is, will people who struggle to make their own bots be poor for the rest of lives or will financial market collapses to it's predictability?

Won't then financial market would be forced to fuck up with all AI to make money, therefore making unpredictable mess nobody of humanity wants?

Anonymous

Anonymous Tue 14 Jan 2020 00:11:24 No.11299947 Report

Quoted By:

>>11299940
np snail brain

Anonymous

Anonymous Tue 14 Jan 2020 00:14:57 No.11299957 Report

Quoted By: >>11299984 >>11300051

>>11299934
That will help too, because it effectively adds 12 data samples per sequence.
Return the whole output sequence from the lstm and do your mse on the sequence, off by one timestep.

Anonymous

Anonymous Tue 14 Jan 2020 00:18:46 No.11299974 Report

Quoted By:

>>11298923
>>11299762

actually the market is very ergodic.

look up
https://en.wikipedia.org/wiki/Robert_Mercer

he was a pioneer in machine learning

Anonymous

Anonymous Tue 14 Jan 2020 00:21:19 No.11299984 Report

Quoted By: >>11299996

>>11299934
>>11299957
do i need to run lstm again on whole 720 long dataset? like take that 720 samples, add to it one predicted sample, use it to predict next 11 and compare? or remove first day from 720 samples, and append to it first predicted day, use to predict next 11 days? I get data from coinpaprika, just convert json to pandas

Anonymous

Anonymous Tue 14 Jan 2020 00:25:56 No.11299996 Report

Quoted By: >>11300051

>>11299984
You say "whole 720 long dataset" like that's big. Speaking of, are you sure you're using the GPU? Your shit is ridiculously slow compare to what it should be for such a minuscule model and dataset.
What you do is give X[:, 0:-1] to the model and have it predict X[1:]. With the fix to your shapes/batch_input_shape, the model will produce one output per timestep and each price is going to correspond to one dimension per timestep. Thus your prediction in total will be of shape batch_size, 11, 1. You want to backprop the mse(pred, X[1:]), basically.
Normally you expect the model to converge in like 30 seconds max by the way.

Anonymous

Anonymous Tue 14 Jan 2020 00:38:31 No.11300036 Report

Quoted By:

>>11298923
No. Impossible.

>>11298968
It's possible to get the data of the last 40 years. I did it with a friend. Had to circumvent some rate limiting restrictions, but it wasn't that hard. It will lead to nothing though.

Best system is Aladdin by BlackRock. They do sentiment analysis on tweets, news and similar pieces of info plus some voodoo stuff. Even they don't know what's going to happen because stock prices are almost completely driven by human psychology.

Anonymous

Anonymous Tue 14 Jan 2020 00:44:09 No.11300051 Report

Quoted By:

>>11299996
>>11299957
>>11299934
>>11299926
>>11299918
>>11299821
thank you for help, tomorrow i am going to read on LSTM and mean reversion so i know whats going on

Anonymous

Anonymous Tue 14 Jan 2020 00:47:44 No.11300062 Report

Quoted By: >>11300095

btw i want to base my thesis on this, do you think its a good idea? Base thesis on something that may not work at all (because some peoples says market is unpredictable)?

Anonymous

Anonymous Tue 14 Jan 2020 00:57:00 No.11300095 Report

Quoted By:

>>11300062
You could, but consider how to phrase it. For example you could evaluate different temporal models (including LSTMs) as market predictors and compare what happens when you add additional information like sentiment analysis. You can then conclude more or less what part of the information can legitimately be ascribed to "organic" market flow vs psychology for example, that could be interesting.

Anonymous

Anonymous Tue 14 Jan 2020 02:15:52 No.11300270 Report

Quoted By: >>11300454

>idiots ITT who don't realize that trying to beat the market is LITERALLY THE HARDEST GAME you can play because everyone else, including people with much more intelligence and resources, are trying to do the same
Why is /sci/ so full of retards?

Anonymous

Anonymous Tue 14 Jan 2020 03:30:12 No.11300454 Report

Quoted By:

>>11300270
Did you even read the thread at all?
Protip: no.

Anonymous

Anonymous Tue 14 Jan 2020 15:36:30 No.11301391 Report

Quoted By:

bump

Anonymous

Anonymous Tue 14 Jan 2020 16:13:24 No.11301462 Report

Quoted By:

>>11299492
>aye my shitcoin guys
>we gonna be rich we gons make it (going on for 6 years)
>a thread about business? Have you even bought my shitcoin?
>what the fuck do you mean a real stock, can I use my ShitCoin to buy it? What is that?
>guys it has lost you money, but NOW is the time to put more into it NOW WITHIN 2H
>oh I'm gonna be a millionaire after putting $5 in the right coin
>I can't help you with tips concerning business, but I will shill you my shitcoin
>no, no it's good business to shitpost day in and out for 5 years making absolutely nothing while a kid in China makes more per hour than me it really is gonna blow up youguys
>>>/biz/

Anonymous

Anonymous Tue 14 Jan 2020 20:42:13 No.11301873 Report

Quoted By: >>11302244

>>11299230
should i use reddit for sentiment analysis? Scraping is very simple with praw library but there could be a lot of shills

Anonymous

Anonymous Tue 14 Jan 2020 20:45:16 No.11301877 Report

Quoted By:

you should know every variable in order to predict it and that's impossible so no

Anonymous

Anonymous Tue 14 Jan 2020 20:51:34 No.11301887 Report

Quoted By: >>11302166

You need to watch the news
even better: create some sort of program that scans every site that has a relevant news words for whatever stock you are looking for and has such words as 'increase' or 'higher than predicted' and add it to a counter

from there it spits out a number that you can use as a baseline from which u should buy or sell that day

this is the exact job of a financial adviser, and that is why they get paid 100k+

Anonymous

Anonymous Tue 14 Jan 2020 20:53:13 No.11301891 Report

Quoted By: >>11302246

ITT: this is how ML retards actually think

Anonymous

Anonymous Tue 14 Jan 2020 22:54:13 No.11302166 Report

Quoted By:

>>11301887
Can I use ML to predict the future ?

Anonymous

Anonymous Tue 14 Jan 2020 23:03:02 No.11302192 Report

Quoted By:

>>11298923
No one who makes money in the stock market is predicting shit.

You don't need ai for the only thing that matters, risk control.

Anonymous

Anonymous Tue 14 Jan 2020 23:29:43 No.11302244 Report

Quoted By:

>>11301873
You could, there are already large reddit datasets available in the wild, as well as many other kinds of datasets. Google around and you'll find them.

Anonymous

Anonymous Tue 14 Jan 2020 23:30:45 No.11302246 Report

Quoted By:

>>11301891
At least they think, unlike your ilk.

Anonymous

Anonymous Tue 14 Jan 2020 23:35:03 No.11302253 Report

Quoted By:

>>11298923
In short, No. It'll do a better job than any human but it'll still be unreliable.

Capcode	All Only User Posts Only Moderator Posts Only Admin Posts Only Developer Posts
Show Posts	All Only With Images Only Without Images
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Ghost Posts	All Only Ghost Posts Only Non-Ghost Posts
Post Type	All Only Sticky Threads Only Opening Posts Only Reply Posts
Results	All Grouped By Threads
Order	Latest Posts First Oldest Posts First

Your latest searches