/sci/ - Science & Math » Thread #11400084

41KiB, 1140x615, Standard-Deviation-Blog.jpg

View Same Google iqdb SauceNAO

Anonymous Wed 19 Feb 22:05:58 2020 No.11400084 View Reply Original Report

Quoted By: >>11400122 >>11400203 >>11400585 >>11400699 >>11400771

i have a dataset that won't fit in RAM. yet, i need to calculate the standard deviation over it. what i could do at this point is get the stddev per file.

then, is there any way to calculate the "global" stddev from many "local" stddevs? (i think there is no way...)

how else can i achieve this?

Anonymous

Anonymous Wed 19 Feb 2020 22:07:42 No.11400092 Report

Quoted By: >>11400116

I think you can upper bound it. If you describe the issue more simply maybe I can help you but I do not understand what it means to fit into the RAM

Anonymous

Anonymous Wed 19 Feb 2020 22:13:28 No.11400116 Report

Quoted By: >>11400170

>>11400092
thanks! by RAM i mean i have many files that make up a dataset, and i would need to load them all in at the same time in order to calculate a mean and a proper stddev (for normalization purposes).
this is impossible due to hardware limits; i can't load all files at once.

instead, i could load one file, get its mean and stddev, then load the next and so on.

then, after obtaining all values, i can't just average the stddevs. i don't know how to calculate the "global" value.

but i think i actually just found a way...
https://math.stackexchange.com/questions/2044757/how-to-calculate-standard-deviation-of-entire-set-from-means-and-populations-of

Anonymous

Anonymous Wed 19 Feb 2020 22:14:23 No.11400122 Report

Quoted By: >>11400142

>>11400084
You fucking negroid, do you think anyone that works with data just has terabytes of RAM laying around? Do you think that any computer at CERN has a petabyte of RAM?

Anonymous

Anonymous Wed 19 Feb 2020 22:18:51 No.11400142 Report

Quoted By: >>11400348

>>11400122
yes i do

hell, there's plenty of ways to do normalization, and there's plenty of debate on which one's best. i have no idea which of these plenty methods Google or CERN or Negroid Inc. follow.

Anonymous

Anonymous Wed 19 Feb 2020 22:19:50 No.11400149 Report

Quoted By: >>11400213

Just break it up into equal size samples, compute the mean for each sample, then compute the mean of the means from the samples to get the mean of the dataset. Calculate and store variance for the dataset, then repeat with samples from the variance to calculate the standard deviation. Simple.

Anonymous

Anonymous Wed 19 Feb 2020 22:24:07 No.11400170 Report

Quoted By: >>11400213 >>11400222

>>11400116
>i would need to load them all in at the same time in order to calculate a mean and a proper stddev
I don't understand why it's necessary to load all files at once to find a global mean or stddev

> instead, i could load one file, get its mean and stddev, then load the next and so on.
Seems to me it'd be better to iterate over all files to find the final sum, then divide to get the global mean.

Same for the stddev. Just iteratively do each calculation, which you'd be doing even if you could load the entire set. But you may just need to break the formula down into smaller problems, solve as much as you can within your constraints and repeat until the larger problem(stddev) has been solved.

Anonymous

Anonymous Wed 19 Feb 2020 22:27:31 No.11400188 Report

Quoted By:

By not fitting in ram, do you mean you use excel or something to load them ? Because you can probably make a python script that loads files one by one and dont saturate your ram

Anonymous

Anonymous Wed 19 Feb 2020 22:28:59 No.11400203 Report

Quoted By:

>>11400084
Is this fucking homework? This is homework isn't it.

Anonymous

Anonymous Wed 19 Feb 2020 22:30:38 No.11400213 Report

Quoted By: >>11400234

>>11400149
>>11400170
ok, while my program iterated over the files, i got their means and variances. after all iterations, i now have:
- mean per file
- the mean of all means
- variance per file

before i go on for the global standard deviation, i will catch up with the maths. i realize i lack some understanding...

Anonymous

Anonymous Wed 19 Feb 2020 22:32:37 No.11400222 Report

Quoted By: >>11400263

>>11400170
Adding everything then dividing can lead to unexpected behavior like overflows if you have a fuckhuge amout of numbers, it may be better to divide each number first then adding them to the sum

Anonymous

Anonymous Wed 19 Feb 2020 22:34:59 No.11400234 Report

Quoted By: >>11400282

>>11400213
The standard deviation is the mean of the variance, braheim. You're basically there already.

Anonymous

Anonymous Wed 19 Feb 2020 22:40:17 No.11400263 Report

Quoted By: >>11400284 >>11400320

>>11400222
Dividing first can lead to rounding or truncation errors. Better use some big number algorithms to perform all the arithmetic.

Anonymous

Anonymous Wed 19 Feb 2020 22:45:45 No.11400282 Report

Quoted By: >>11400329

>>11400234
No? It's the root.

Anonymous

Anonymous Wed 19 Feb 2020 22:46:02 No.11400284 Report

Quoted By:

>>11400263
Yeah, a dedicated library would be the best

Anonymous

Anonymous Wed 19 Feb 2020 22:53:08 No.11400320 Report

Quoted By:

Surely you could just sum all elements, count up the number, divide and get the global mean?
Once that's done, iterate again, add up the square of the element-mean difference, divide by N and square-root at the end.
Just make sure to use high-precision arithmetic to not end up in bad rounding like >>11400263 suggests

Anonymous

Anonymous Wed 19 Feb 2020 22:56:14 No.11400329 Report

Quoted By:

>>11400282
I misspoke, root of the mean of the variance. Whatever though, it's not like it's my homework.

Anonymous

Anonymous Wed 19 Feb 2020 23:01:40 No.11400348 Report

Quoted By:

>>11400142
Hadoop.

Anonymous

Anonymous Wed 19 Feb 2020 23:03:45 No.11400356 Report

Quoted By:

If the dataset really is that huge, then you can easily take a random sample from it that can fit and calculate stddev on it, you’ll get the same number.

Anonymous

Anonymous Thu 20 Feb 2020 00:40:17 No.11400585 Report

Quoted By:

>>11400084
Stream the data to aggregate necessary stats.
>pass 1
compute mean, all you need is K pages + size of the float, and K is variable.
>pass 2
compute sum of squared differences to mean (same requirement as above).
Finally, return the sqrt of that value.

Pretty sure you can even do it in one pass with a little more cleverness.

Anonymous

Anonymous Thu 20 Feb 2020 00:44:51 No.11400593 Report

Quoted By:

Maybe we can have 10 more posters posting the same answers...

This thread should be closed

Anonymous

Anonymous Thu 20 Feb 2020 01:25:28 No.11400699 Report

Quoted By:

>>11400084
What is a "stream" for 200 Alex

Anonymous

Anonymous Thu 20 Feb 2020 01:48:33 No.11400771 Report

Quoted By: >>11400776 >>11400837

>>11400084
Running Variance
https://www.johndcook.com/blog/standard_deviation/

Anonymous

Anonymous Thu 20 Feb 2020 01:50:50 No.11400776 Report

Quoted By:

>>11400771
Actually, the next page has the code for "adding" two stats objects:
https://www.johndcook.com/blog/skewness_kurtosis/

Anonymous

Anonymous Thu 20 Feb 2020 02:11:04 No.11400828 Report

Quoted By: >>11400841

>He doesn't know how to calculate aggregate stats from a data stream

lmao

Anonymous

View Same Google iqdb SauceNAO file.png, 102KiB, 250x250

Anonymous Thu 20 Feb 2020 02:13:55 No.11400837 Report

Quoted By: >>11401873

>>11400771

Anonymous

Anonymous Thu 20 Feb 2020 02:14:53 No.11400841 Report

Quoted By:

>>11400828
>doesn't read

lmao

Anonymous

Anonymous Thu 20 Feb 2020 13:23:11 No.11401873 Report

Quoted By:

>>11400837
It’s like he knows

Capcode	All Only User Posts Only Moderator Posts Only Admin Posts Only Developer Posts
Show Posts	All Only With Images Only Without Images
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Ghost Posts	All Only Ghost Posts Only Non-Ghost Posts
Post Type	All Only Sticky Threads Only Opening Posts Only Reply Posts
Results	All Grouped By Threads
Order	Latest Posts First Oldest Posts First

Your latest searches