The standard error is pretty simple:

${\sigma}_\bar{x}\ = \frac{\sigma}{\sqrt{n}}$

Let's say I take 10 sample IQ scores from the population, and repeat this 5 times, I end up with the following data.

Sample 1 mean: 101.2
Sample 2 mean: 104.2
Sample 3 mean: 106.2
Sample 4 mean: 106.4
Sample 5 mean: 96.6

This is the sampling distribution. The standard deviation of this data is 4.105, and the sample size is 5. So, applying to formula above, gets me this

${\sigma}_\bar{x}\ = \frac{4.105}{\sqrt{5}}=1.836$

But this formula does not take into account the sample size of the samples. In this case, I sampled 10 IQ scores, but I could have sampled 2, or even 1, and the numbers come out the same. Am I missing something?