/sci/ - Science & Math » Thread #12711888

21KiB, 720x287, 20210216_101017.jpg

View Same Google iqdb SauceNAO

Is the formula for calculating the standard deviation wrong?

Anonymous Tue 16 Feb 09:20:11 2021 No.12711888 View Reply Original Report

Quoted By: >>12711900 >>12711904 >>12711922 >>12711927 >>12712085 >>12712222 >>12712262 >>12713282 >>12713771 >>12714227

I was using the standard deviation formula, because it was a part of a homework assignment, when I remembered that the square root of the sum of squared numbers isn't always equal to the sum of the numbers.
I'm quite sure that the reason for squaring the numbers, and then take the square root of them, is so that there are no negative numbers. However, if they wanted to do that, they could've just used abs().
I've tried solving a problem using logic, and it didn't match the result I got by using the formula.
Try finding the standard deviation of (1, 2, 3).
I think we can all agree that the average deviation from the mean is 2/3. Now try to use the formula.
Is there something I'm missing, or is the formula wrong?

Anonymous

Anonymous Tue 16 Feb 2021 09:24:26 No.12711900 Report

Quoted By:

>>12711888
You're thinking of variance

Anonymous

Anonymous Tue 16 Feb 2021 09:26:08 No.12711904 Report

Quoted By: >>12713477

>>12711888
read your fucking book, and this is an 18+ board

Anonymous

Anonymous Tue 16 Feb 2021 09:34:31 No.12711922 Report

Quoted By:

>>12711888
>However, if they wanted to do that, they could've just used abs().
that is the L1-norm or something and it is awkward to work with analytically

Anonymous

Anonymous Tue 16 Feb 2021 09:38:04 No.12711927 Report

Quoted By: >>12712254

>>12711888
>I'm quite sure that the reason for squaring the numbers, and then take the square root of them, is so that there are no negative numbers
no
>the square root of the sum of squared numbers isn't always equal to the sum of the numbers.
that's the fucking point. compared to a mere sum, sqrt of sum of squares is negligible for x_i close to mu, but grows very large when x_i is far from mu. that's exactly what the deviation is supposed to capture.

Anonymous

Anonymous Tue 16 Feb 2021 11:11:35 No.12712085 Report

Quoted By: >>12712225 >>12712256

>>12711888
Yeah it's wrong. You should divide by (n-1) instead.

Anonymous

View Same Google iqdb SauceNAO 1609535192615.jpg, 141KiB, 1024x684

Anonymous Tue 16 Feb 2021 13:00:10 No.12712222 Report

Quoted By:

>>12711888
i adore your enthusiasm but srsly do you even know what you argue for?
If you are inetrested in understanding this in wider context other than being given a formula with some intuition you need to work through introductory probability and mathematical stats- learn about random variables, variance, mean of a random variable and their estimators.
It's often the case in the introductory courses you are learning about descriptive statistics and you basically just memorize formulas, and learn more logic about it in later courses.

Anonymous

Anonymous Tue 16 Feb 2021 13:04:43 No.12712225 Report

Quoted By: >>12712231

>>12712085
Nah. That's only when N is small

Anonymous

Anonymous Tue 16 Feb 2021 13:09:26 No.12712231 Report

Quoted By:

>>12712225
N=3 is small.

Anonymous

Anonymous Tue 16 Feb 2021 13:22:34 No.12712254 Report

Quoted By: >>12712265

>>12711927
>negligible for x_i close to mu, but grows very large when xi isfar from mu
this right here, that's the key. we want to use the power of 2 to emphasize large deviations from the mean. the square root is just so you dont end up with stupidly large numbers all over the place but want to preserve the effects of that squaring.
having every deviation follow a linear "weighing" makes it harder to analyze large spreads of data, otherwise maybe it would be the standard by now???

Anonymous

Anonymous Tue 16 Feb 2021 13:22:49 No.12712256 Report

Quoted By:

>>12712085
depends on if you want the sample variance or not
if the entire population only contains 3 elements then well

Anonymous

Anonymous Tue 16 Feb 2021 13:26:47 No.12712262 Report

Quoted By:

>>12711888
I appreciate that you're thinking about different measures of spread of a distribution, but the reality is that variance / standard deviations (which is a root-mean-square of the errors) is just much more well-behaved than other sorts of measures. You could average the absolute errors and call that the spread. You could take the maximum absolute error and call that the spread (this is a terrible idea when you start having continuous probability distributions). You could average the pth powers of the absolute errors and then take the pth root for any p > 1. But none of these work as nicely and have as many great properties as the standard deviation (the case p=2). You see this a lot in fields like probability, analysis, and measure theory. It's related to the fact that in Euclidean space, the length of a vector is given by the Pythagorean theorem. Think as though you have an "error vector" and you want to find its length, and then to normalize by the number of terms in the vector.

Anonymous

Anonymous Tue 16 Feb 2021 13:28:56 No.12712265 Report

Quoted By: >>12712349 >>12713456 >>12713527

>>12712254
Linear weighting is a mess in many ways. Yes, what you highlight is essential to several properties that the variance / std dev obeys, but it's not really the core reason. The core reason is that sums of squares act better and more geometrically than sums of absolute values.

Anonymous

Anonymous Tue 16 Feb 2021 13:53:01 No.12712349 Report

Quoted By:

>>12712265
ditto, thanks for elaborating fren

Anonymous

Anonymous Tue 16 Feb 2021 19:40:17 No.12713247 Report

Quoted By:

>they could've just used abs().

But did not. Its human consept.

Anonymous

View Same Google iqdb SauceNAO FAE_visualization.jpg, 85KiB, 700x525

Anonymous Tue 16 Feb 2021 19:42:16 No.12713256 Report

Quoted By:

Dont blame OP. This shit is not told even somewhat advanced literature. I was lucky in my language there was a book written by a real probability mathematican and this was discussed.

The reasoning is that abs() is harder to calculate, so take the naive way out.

Anonymous

View Same Google iqdb SauceNAO CodeCogsEqn(1).gif, 846B, 154x42

Anonymous Tue 16 Feb 2021 19:49:21 No.12713278 Report

Quoted By:

Anonymous

Anonymous Tue 16 Feb 2021 19:50:08 No.12713282 Report

Quoted By:

>>12711888
OP, you're right and you're not the first to notice it. There are a lot of pseuds on this thread that feel threatened by having their math challenged. I've thought about this problem and I've come to three main conclusions on it.
1. Using the abs() function is, pun intended, absolutely valid here. There is nothing preventing us from using it for standard deviation.
2. There is also nothing stopping us from calculating standard deviation using the root mean square, as it's also a proxy to the average. We use this all the time in electronic signals with RMS Voltage.
3. If forced to choose, there are several criteria you can use to make a selection. Mathematically, it's easier to handle root mean squares than it is to handle absolute values. This is due to smoothness of the function.
Ultimately, the conclusion I came to is that philosophically we should be using abs() for error calculations. Firstly it just makes more intuitive sense to measure spread. Secondly, from a philosophy perspective, we should always report the maximum possible error in our measurements. Linear propagation of error using abs() has a larger error than the quadratic one. Thirdly, as for all the arguments for why we need a "cleaner" function analytically for error, I've never seen one concrete example for this. It's hand waving dismissal that's supposed to be the domain of physicists, but I find it shocking that mathematicians who praise each other for rigor cannot provide a single rigorous, concrete example for why we need this cleaner analytic version of error propagation.

Anonymous

View Same Google iqdb SauceNAO CodeCogsEqn(6).gif, 1KiB, 200x53

Anonymous Tue 16 Feb 2021 19:50:46 No.12713283 Report

Quoted By:

Anonymous

Anonymous Tue 16 Feb 2021 20:39:15 No.12713456 Report

Quoted By: >>12713527

>>12712265
>more geometrically than sums of absolute values.
I'm curious what you mean by that. I would've said the linear weighting thing + all the favorable properties that working with moments in general lead to, which MAE doesn't.

Anonymous

Anonymous Tue 16 Feb 2021 20:44:52 No.12713477 Report

Quoted By: >>12714586

>>12711904
The first time I used this equation was in a stats class in college

Anonymous

Anonymous Tue 16 Feb 2021 21:01:44 No.12713527 Report

Quoted By: >>12713556 >>12714035 >>12714882

>>12712265
>>12713456
I think the core reason is that *if* the values are normally distributed, then the sum of squared deviations is preferred because stdev/variance is the spread parameter in the definition of the Gaussian distribution. In contrast, the Laplace distribution uses the sum of absolute deviations to parameterize it's spread. Both have their uses, with the (lesser known) Laplace distribution having applications in Bayesian statistics and machine learning.

Anonymous

View Same Google iqdb SauceNAO Gaussian-distribution-and-Laplac (...).png, 77KiB, 850x640

Anonymous Tue 16 Feb 2021 21:09:26 No.12713556 Report

Quoted By: >>12713713

>>12713527
pic related

Anonymous

Anonymous Tue 16 Feb 2021 22:03:32 No.12713713 Report

Quoted By:

>>12713556
chad Laplace vs virgin Gauss

Anonymous

Anonymous Tue 16 Feb 2021 22:24:55 No.12713771 Report

Quoted By:

>>12711888
Variance is expectation of x-squared minus expectation of x, squared

?^2 = ?(x-?)^2/N - (?(x-?)/N)^2

Anonymous

Anonymous Tue 16 Feb 2021 23:48:22 No.12714035 Report

Quoted By: >>12714369

>>12713527
Yeah that makes sense. What do you use the Laplace dist for? I do Bayesian stats all the time (albeit with domain specific nonlinear models) and haven’t ever seen a reason to. Is it conjugate/analytically convenient to some weird likelihood when you’re working with discrete Markov chain models or something?

Anonymous

Anonymous Wed 17 Feb 2021 00:37:45 No.12714227 Report

Quoted By:

>>12711888
>I think we can all agree that the average deviation from the mean is 2/3. Now try to use the formula.
>Is there something I'm missing, or is the formula wrong?
That is why it is called standard deviation and not average deviation.
>However, if they wanted to do that, they could've just used abs().
Abs would give you the L1 "distance" from the mean.
In some sense it is harder to work with. It does make things like Jensen's inequality applicable, though (you could use |x|^p for any p>=1).
https://en.wikipedia.org/wiki/Jensen%27s_inequality
One main reason the L2 (sum of squares) is used is because it is a "cleaner" way to make things positive.
It also guarantees the mean, m, is the value that minimizes it. If you write sum{(x_i - m)^2} then differentiate with respect to m to find the minimum, you find it is the mean that gives the minimum.
A similar trick of minimizing the L2 shows up in regression.
When doing regression, your data are points (x_i, y_i) and you wish to find a linear function f(x) = a*x+b = y that minimizes your "distance" sum{d(f(x_i),y_i))}.
Choosing d to be the L2 metric gives you a linear equation after taking the gradient with respect to a and b which is easily and exactly solvable.

Anonymous

Anonymous Wed 17 Feb 2021 01:16:41 No.12714369 Report

Quoted By:

>>12714035
Compared to a Gaussian, the Laplace distribution has greater concentration at 0 and wider tails. This is a useful property for promoting sparsity and robustness. As others have pointed out, it's related to the L1 norm.
In a Bayesian setting, the robust lasso regression is equivalent to placing a Laplace prior on the error term.

Anonymous

Anonymous Wed 17 Feb 2021 02:46:44 No.12714586 Report

Quoted By:

>>12713477
kek brainlet

Anonymous

Anonymous Wed 17 Feb 2021 04:43:50 No.12714882 Report

Quoted By:

>>12713527
Additionally, the concept of measuring spread of errors go hand in hand with the Gaussian distribution in introductory classes for numerous reasons. Since the model very plainly includes variance already, standard deviation can be defined in relation to the models definition rather than taking time to redefine it in other terms like absolute sums.

Also OP, standard deviation is a fine metric for spread, but variance is better to work with. It's a bit easier to go into ordered moments using variance as your coner stone of population error measurements, variances can be additive, has a tighter relationship to covariance, etc. I think the definition of standard deviation being in close agreement to variance in normal models has some strong advantages when being introduced to statistics and probability

Anonymous

Anonymous Wed 17 Feb 2021 16:20:18 No.12716614 Report

Quoted By:

bump

Capcode	All Only User Posts Only Moderator Posts Only Admin Posts Only Developer Posts
Show Posts	All Only With Images Only Without Images
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Ghost Posts	All Only Ghost Posts Only Non-Ghost Posts
Post Type	All Only Sticky Threads Only Opening Posts Only Reply Posts
Results	All Grouped By Threads
Order	Latest Posts First Oldest Posts First

Your latest searches