/sci/ - Science & Math » Thread #12541107

14KiB, 300x168, download.jpg

View Same Google iqdb SauceNAO

Anonymous Sun 03 Jan 20:06:58 2021 No.12541107 View Reply Original Report

Quoted By: >>12541338 >>12541556

I have a question for stats experts here:

What is a viable test of significance to use for non-normal data in the scenario below?

I want to test whether sample results from 35 trials (derived from randomized permutations of an algorithm) differ significantly from an expected output value.

The data is not normal. Output might look like this:

Expected value: 4.7

Sample N: 35
Sample mean: 4.714
Sample stdev: 0.043
where 31 samples yield 4.7, 3 cases yield 4.8, and 1 case yields 4.9

Any thoughts or help is much appreciated. Thanks!

Anonymous

Anonymous Sun 03 Jan 2021 20:55:09 No.12541283 Report

Quoted By:

Just bumping, friends

Anonymous

Anonymous Sun 03 Jan 2021 21:07:31 No.12541338 Report

Quoted By: >>12541346 >>12541434

>>12541107
>I want to test whether sample results from 35 trials (derived from randomized permutations of an algorithm) differ significantly from an expected output value.
to clarify, you are testing whether the mean of the samples are the same as an expected value mean?
If so, I'd normally use a 1-sample t-test and be done with it; you have enough samples for the normality assumption to hold.
But if you want to take into account the non-normality of your data, one sample wilcoxon signed rank test is a non-parametric one-sample test you can use
Or bootstrap. I love bootstrapping.

Anonymous

Anonymous Sun 03 Jan 2021 21:09:29 No.12541346 Report

Quoted By:

>>12541338
>you have enough samples for the normality assumption to hold.
I mean you have enough samples that normality shouldn't matter that much. you'll still have some inflated type-something error, but probs not enough to matter.

Anonymous

Anonymous Sun 03 Jan 2021 21:23:47 No.12541434 Report

Quoted By: >>12541505

>>12541338
I’m quality testing a scoring algorithm that yields a single expected value. When the text that the algorithm scores is randomly modified, the scoring assessment yields a potentially different value, though it usually seems to spit out the expected value. Without going into too much detail, I’m evaluating the validity of scoring constructs.

So the data could still be normal even if it doesnt seem as though it is?

Anonymous

Anonymous Sun 03 Jan 2021 21:38:54 No.12541505 Report

Quoted By: >>12541545

>>12541434
More clarification since I'm not 100% sure I am answering your question:
What do you want the test to tell you? In you example, If you want the test to tell you that the mean of 35 trials is different from the expected value mean of 4.7, then wilcoxon signed rank test away. But I'm not sure that's exactly what you want the test to tell you.
>So the data could still be normal even if it doesnt seem as though it is?
No, just that thanks to the central limit theorem, once you have enough samples, it doesn't matter what your distribution of your data is. The normality assumption actually has to do with the distribution of means if you were to pull a random sample, calculate a mean; pull another sample, calculate another mean; when your samples are large enough, the distribution of means becomes normal no matter the shape of your actual data distribution. Rule-of-thumb is >30, but I can tell you its wishy-washy ish.

Anonymous

Anonymous Sun 03 Jan 2021 21:48:39 No.12541545 Report

Quoted By:

>>12541505
That’s super helpful, much appreciated.

So the scoring algorithm analyzes a text, and so long as the text isn’t modified, it will always spit out the same score (4.7 in the case of this particular text). The latent constructs (as defined by composition theory) should expect that the scoring of this text will change in a manner that is significant when the text is randomized, but I am hypothesizing that the changes are not significant (i.e., the algorithm does not measure what it claims to measure according to theory) . So yes, I am seeking to identify if the mean of a random sample of modifications is significantly different than the only expected value (which could be thought of as an expected mean since even a million iterations of the algorithm always yields the same score so long as the text is unchanged)

Anonymous

Anonymous Sun 03 Jan 2021 21:50:45 No.12541556 Report

Quoted By: >>12541580

>>12541107
Means of your samples should still be normally distributed so you can do a regular t-test like some other anon mentioned

Anonymous

Anonymous Sun 03 Jan 2021 21:56:32 No.12541580 Report

Quoted By: >>12541618

>>12541556
Ooooohhhhh, so normality is all about the distribution of randomly selected sample means, not the mean of the actual values themselves? Lol stats blows my mind.

Anonymous

Anonymous Sun 03 Jan 2021 22:02:51 No.12541618 Report

Quoted By:

>>12541580
Well it's just that the variance of a group of means is asymptotically normal.
You can verify this yourself by checking the variance of the sum of a r.v. from your special distribution divided by the the number of samples.

Capcode	All Only User Posts Only Moderator Posts Only Admin Posts Only Developer Posts
Show Posts	All Only With Images Only Without Images
Deleted Posts	All Only Deleted Posts Only Non-Deleted Posts
Ghost Posts	All Only Ghost Posts Only Non-Ghost Posts
Post Type	All Only Sticky Threads Only Opening Posts Only Reply Posts
Results	All Grouped By Threads
Order	Latest Posts First Oldest Posts First

Your latest searches