I like your example where the complications of non-parametric tests are stripped away. Let me be specific in the context of this example. Because each system (and each pair of systems) is different, every point has a different Var[d(one unit)]. Thus, the normal-cdf used to compute the p-value will be different for every point on the plot. Therefore the “curve-shaped trends” (I’ll admit this bigram was overused in the paper) we see are not the result of a basic statistical fact as you claim. Instead, they tell us that in the region we care about (i.e. near 0.05) the effects of system variation are dominated by the effects of test set size, and as a result we CAN loosely treat these plots as though they arise from a single cdf. This may not be particularly surprising, but it’s also not obvious a priori. By the way, we do find thresholds that loosely imply significance. This is contrary to what you wrote in your blog. Not sure if that was a typo.

Anyway, I do think your points about complicated metrics are interesting. Something that you may already know: BLEU is asymptotically normal, if you ignore the single discontinuity in the derivative of the brevity penalty. You can prove this with Slutsky’s theorem and the delta method. So perhaps a parametric test is just fine for BLEU when the test set is large.