Quoted By:
Let's say you've got a linear regression model with all the standard OLS assumptions. Your estimate is a random variable drawn from a normal distribution centered around the true value. One natural statistic is the t-statistic, which is essentially a normalized measure of the distance between the estimate and the null hypothesis value (H0). Because the estimate is random, so is the t-statistic. If you don't know the true standard error and replace it by an unbiased estimator of the standard error, the t-statistic follows a Student distribution. If the null is correct, the t-statistic follows a centered Student distribution. Otherwise, the t-statistic follows a non-centered Student distribution. In this case, the p-value is the probability of the statistic value being at least as large as it is (in absolute value) if it follows a centered Student distribution compatible with the null. The lower the p-value, the more confident you can be that your t-statistic was drawn from another distribution than one compatible with the null. If your t-statistic is associated with a p-value of 0.001, you are saying there is only a 0.1% chance of witnessing a t-statistic at least this far from 0 if the null hypothesis is true.
The picture is a visual representation of a t-test at level 5%. The red curve is the null distribution. The black bars are the cut-off points for the t-test associated with a 0.05 p-value. The blue histogram is random draws of the t-statistic from the true distribution. In practice, you don't know the true distribution and can only ever see one draw from it. When your t-statistic lands beyond the black bars, you are tempted to reject H0 because you think the t-statistic must have come from another distribution than one compatible with the null. You might still be wrong because the red curve still has a density beyond the black bars, but the evidence weighs heavier on the side of the t-statistic having come from another distribution.
