/sci/ - Science & Math » Thread #14151583

54KiB, 600x309, Screenshot 2021-09-04 at 01.19.27.png

View Same Google iqdb SauceNAO

Anonymous Wed 26 Jan 16:40:10 2022 No.14151583 View Reply Original Report

Quoted By: >>14151710 >>14151807 >>14151814 >>14151931 >>14151935 >>14152613 >>14152727 >>14153007 >>14153538 >>14153756 >>14153984 >>14155083

how do you handle extreme values in out of sample data messing up your statistical models?

How do I account for tail-end events? Is the only way to get more data?

Anonymous

Anonymous Wed 26 Jan 2022 17:19:08 No.14151710 Report

Quoted By: >>14151814 >>14151948 >>14151985 >>14152601 >>14153984

>>14151583
Outliers should be removed and hidden away if they don't match up with your hypothesis. That's the proper academia procedure.

Anonymous

Anonymous Wed 26 Jan 2022 17:21:14 No.14151716 Report

Quoted By: >>14151814

Do more tests

Anonymous

Anonymous Wed 26 Jan 2022 17:57:09 No.14151807 Report

Quoted By: >>14151814 >>14151848 >>14151989

>>14151583
If outliers fuck up your model, then you don't have enough data.

Anonymous

View Same Google iqdb SauceNAO 1619046238575.jpg, 37KiB, 400x400

Anonymous Wed 26 Jan 2022 17:59:43 No.14151814 Report

Quoted By: >>14151827

>>14151583
>>14151716
>>14151807
Actually take a fucking statistics course, like this guy did:
>>14151710

Anonymous

Anonymous Wed 26 Jan 2022 18:05:43 No.14151827 Report

Quoted By:

>>14151814
I just shat my pants laughing bro

Anonymous

Anonymous Wed 26 Jan 2022 18:11:04 No.14151848 Report

Quoted By: >>14151889 >>14152608 >>14153812

>>14151807
It's not more possible to get more data - what else can I do?

Anonymous

Anonymous Wed 26 Jan 2022 18:23:52 No.14151889 Report

Quoted By:

>>14151848
I suppose if you know for a fact they are outliers, remove them from your dataset. Not much else you can do.

Anonymous

Anonymous Wed 26 Jan 2022 18:38:24 No.14151931 Report

Quoted By: >>14152503

>>14151583
What kind of test or inference are you doing?

Anonymous

Anonymous Wed 26 Jan 2022 18:39:30 No.14151935 Report

Quoted By: >>14151945 >>14151962

>>14151583
What the fuck kinda ugly ass language is that?

Anonymous

Anonymous Wed 26 Jan 2022 18:42:30 No.14151945 Report

Quoted By: >>14151962 >>14152634

>>14151935
Looks like possibly JavaScript? Fucking repugnant isn't it.

Anonymous

Anonymous Wed 26 Jan 2022 18:44:06 No.14151948 Report

Quoted By:

>>14151710
Holly based.

Anonymous

Anonymous Wed 26 Jan 2022 18:49:47 No.14151962 Report

Quoted By: >>14152634

>>14151945
>>14151935
It's Solidity, a statically typed domain specific language based on JavaScript for creating scams on the Etherum blockchain

Anonymous

Anonymous Wed 26 Jan 2022 18:57:53 No.14151985 Report

Quoted By: >>14152512

>>14151710
>Outliers should be removed and hidden away if they don't match up with your hypothesis. That's the proper academia procedure.
This. Make sure to come up with some bullshit excuse that "justifies" removing them. "The value is three standard deviations above the mean, while the rest of the data has a highly leptic normal distribution"; "it is the only value belonging to this octant, strengthening our conviction that this datapoint is a result of measurement error", something like that.

Anonymous

View Same Google iqdb SauceNAO 1641956380341.jpg, 12KiB, 139x175

Anonymous Wed 26 Jan 2022 18:59:01 No.14151989 Report

Quoted By: >>14152582

>>14151807
or your model is wrong

Anonymous

Anonymous Wed 26 Jan 2022 21:20:52 No.14152503 Report

Quoted By: >>14152524 >>14152600

>>14151931
Running the model on a validation dataset then getting the rmse and mae. There are 20-30 large outliers in the validation data which cause the model to have a really bad validation score.

Is there a way to stabilise the model? The outliers are somewhat clustered together but that is the only thing I can be sure about.

Anonymous

Anonymous Wed 26 Jan 2022 21:24:22 No.14152512 Report

Quoted By:

>>14151985
>this data was invented by Robert Malone
No need to say more.

Anonymous

Anonymous Wed 26 Jan 2022 21:27:21 No.14152524 Report

Quoted By: >>14152555

>>14152503
if those 20-30 outliers are easily distinguishable from the other then just remove them. And say that your model is not valid for outliers.

Anonymous

Anonymous Wed 26 Jan 2022 21:36:57 No.14152555 Report

Quoted By: >>14152567 >>14152600 >>14153823

>>14152524
If I deploy this into production and know that it will categorically fail on tail-end events then I need a way to hedge that risk. How can I do that? The hedging has to be external to the model?

Anonymous

Anonymous Wed 26 Jan 2022 21:40:06 No.14152567 Report

Quoted By:

>>14152555
Oh I realise I’m using the wrong terms here… the statistics term is “regularisation” not “hedging”

Anonymous

Anonymous Wed 26 Jan 2022 21:44:07 No.14152582 Report

Quoted By:

>>14151989
Or you fucked up a measurement.

Anonymous

Anonymous Wed 26 Jan 2022 21:48:53 No.14152600 Report

Quoted By:

>>14152503
>>14152555
You cannot even explain what you are doing. Build a model from first principles and take the precision you get.
Think like Box: All models are wrong, but some are useful. Is yours?

Anonymous

View Same Google iqdb SauceNAO infront-of-ben-shapiro-70982030.png, 108KiB, 500x481

Anonymous Wed 26 Jan 2022 21:49:02 No.14152601 Report

Quoted By:

>>14151710
scientists love talking about diversity yet they don't respect diversity in their data, curious

Anonymous

Anonymous Wed 26 Jan 2022 21:50:34 No.14152608 Report

Quoted By:

>>14151848
thompsons T test
basic analytical method

Anonymous

Anonymous Wed 26 Jan 2022 21:51:32 No.14152613 Report

Quoted By:

>>14151583
Idk...
I'd remove them under the assumption they are errors. But include a separate run somewhere, that assumes they aren't errors for contrast and in case they aren't actually errors.

Anonymous

Anonymous Wed 26 Jan 2022 21:57:47 No.14152634 Report

Quoted By:

>>14151962
>a statically typed domain specific language based on JavaScript for creating scams on the Etherum blockchain
Please remove me from this timeline already. Im just a math autist yameroooo

>>14151945
Absolutely sickening

Anonymous

Anonymous Wed 26 Jan 2022 22:00:06 No.14152640 Report

Quoted By: >>14152959

What meme programming language is this?

Anonymous

Anonymous Wed 26 Jan 2022 22:39:28 No.14152727 Report

Quoted By:

>>14151583
If it's an extreme outlier in an otherwise clear trend - leave it in the plot but exclude it from the fit.

Anonymous

Anonymous Thu 27 Jan 2022 00:03:19 No.14152959 Report

Quoted By: >>14152971

>>14152640
Solidity, actually one of the more useful programming languages.
Made 15k in a week off of it.

Anonymous

Anonymous Thu 27 Jan 2022 00:07:19 No.14152971 Report

Quoted By:

>>14152959
>blockchain language
Meme status confirmed

Anonymous

Anonymous Thu 27 Jan 2022 00:23:39 No.14153007 Report

Quoted By:

>>14151583
You can remove outliers if you manually inspect them and find a valid reason your model did not work.
Like a benchmark run during the daylight savings time giving the wrong time or a camera image taken before the blend was removed.

anonymous

anonymous Thu 27 Jan 2022 03:25:31 No.14153538 Report

Quoted By:

>>14151583 Use the median instead of the mean for central tendency. Median is less affected by outliers than median.

Anonymous

Anonymous Thu 27 Jan 2022 04:42:37 No.14153756 Report

Quoted By:

>>14151583
It depends why they're extreme. Recent paper had a bunch of errors in one of the datasets (0 for observations that could not be 0) so we just dropped those points entirely.

Anonymous

Anonymous Thu 27 Jan 2022 05:15:57 No.14153812 Report

Quoted By: >>14154284

>>14151848
You are modeling a statistical distribution, remove the outlier or your findings will be useless.

What to do going forward? Examine the outlier in context. Is there a unrecorded feature that explains it? Update your model.

Anonymous

Anonymous Thu 27 Jan 2022 05:20:16 No.14153823 Report

Quoted By:

>>14152555
Have you tried examining the relationship between your predictors and the occurrence of a tail end event?

Anonymous

Anonymous Thu 27 Jan 2022 06:44:35 No.14153984 Report

Quoted By:

>>14151583
>>14151710
gonna echo what others are saying, you need to investigate why you are getting outliers, is it bad design, a botched experiment or are you getting a real result you didnt expect that shouldnt be ignored

Anonymous

Anonymous Thu 27 Jan 2022 09:48:48 No.14154284 Report

Quoted By: >>14155059

>>14153812
This. You can’t just remove the outliers like anons are saying because then your model is not really useful anymore.

Anonymous

Anonymous Thu 27 Jan 2022 15:39:29 No.14155059 Report

Quoted By:

>>14154284
I'm saying that he should remove the outlier. He's trying to model an underlying statistical distribution and the inclusion of the outlier will skew that approximation.

If his concern is with edge cases then he should model for that instead.

Anonymous

Anonymous Thu 27 Jan 2022 15:49:42 No.14155083 Report

Quoted By:

>>14151583
I wonder could you measure the goodness of a hypothesis by observing the change in estimated parameters when you omit a data point. For better models, the estimated parameter changes little when a random data point is omitted.

Capcode	All Only User Posts Only Moderator Posts Only Admin Posts Only Developer Posts
Show Posts	All Only With Images Only Without Images
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Ghost Posts	All Only Ghost Posts Only Non-Ghost Posts
Post Type	All Only Sticky Threads Only Opening Posts Only Reply Posts
Results	All Grouped By Threads
Order	Latest Posts First Oldest Posts First

Your latest searches