Why does a large sample size cause a significant ANOVA F-test?

If I have a large sample size, e.g. 100,000 data points, I know that most significance tests are going to come back with a very small p-value unless the null hypothesis is "true on the nose." In other words, even very small effects will be seen by the test. I can understand why this is true for a t-test, since when I compute the test statistic I have to divide by $\sqrt$ in the formula for the standard error, so when $n$ is large my standard error is small, and so my t-statistic is huge. Is there a similar explanation for why an ANOVA F-test (let's say 1-way ANOVA) is likely to be significant when $n$ is large? I'm asking so I can better explain things to my Stat 2 class. When asked in class today, the explanation I tried was that, when $n$ is huge $MSE$ is going to be very small (because it's $SSE/(n-k)$), so the $F$-statistic will be huge. The students followed up by asking why the large df in the $F$-statistic doesn't account for this and so give reasonable $p$-values even for very large $F$-statistics (rather than ultra small $p$-values as we've been seeing in our examples). I know, of course, that for a two-sample t-test $F = t^2$, so I can deduce significance as a special case of the reasoning above, but I'm more interested in the general case of more than 2 groups, and an explanation that doesn't require the derivation that $F = t^2$. Any help would be much appreciated. Thanks!

asked Mar 31, 2016 at 17:21 David White David White 267 3 3 silver badges 11 11 bronze badges

$\begingroup$ This question, generalized and posed slightly differently, appears at stats.stackexchange.com/questions/2516. The common spirit is to ask why having more data gives one more power to reject a false null hypothesis. So, rather than focusing on the F-test itself, you might consider discussing this general issue: your students might learn much more for the same effort. $\endgroup$

Commented Mar 31, 2016 at 18:16

$\begingroup$ @whuber. I read that thread just before posting, but could not distill from it an explanation that would satisfy my students. Still, it inspired me to prepare a class on effect size that I'm delivering tomorrow. By the way, I'm a big fan of your answers here. Thanks for posting! $\endgroup$

Commented Mar 31, 2016 at 22:59

$\begingroup$ the explanation I tried was that, when n is huge MSE is going to be very small Sure, you're dividing by a huge $n$, but you're also getting a huge SSE. $\endgroup$

Commented Jan 16 at 20:05

$\begingroup$ Yep, that was literally the next sentence in the question "The students followed up by asking. " Also, the third (last) comment I left below Michael's answer. $\endgroup$