blog - steve t. piantadosi

This situation came up in Rebecca’s class last Wednesday. Some students had gathered data that, looked something like this made up version:

This shows for each subject, their RTs on two conditions of the experiment (Green/Blue). If you throw this sucker into the paired t-test, you find that the difference between conditions is, in fact, not significant (t = -1.8994, df = 10, p > 0.08), even though the difference in means is large (-1.28), and every subject shows the correct trend. But the signed-rank test comes out (p < 0.003).

As Talia pointed out, there’s one subject, the first one, which has a much bigger effect size than the others, and the t-test takes this into account. And, if you remove this first subject, the T-test comes out significant (t = -5.8054, df = 9, p < 0.0005). This is because the t statistic takes cares about the variance in the differences (for each subject) between conditions. If you leave in the first subject, this variance is large and the t statistic decreases in magnitude.

So adding a data point with the correct trend can actually decrease your t statistic, and make this difference nonsignificant. Why is this counterintuitive? Because the first point is a much bigger effect than the others in the right direction! And removing it actually decreases the mean difference between conditions (to -0.62).

Beware!

David told me a good one last night. Here it is:

Suppose that there is a machine which can perfectly predict the future. It is never wrong. You get to play the following game with it. Two boxes are in a room and you can come into the room and open either one or both boxes. Yesterday, the machine predicted whether you would open one or both boxes, but you don’t know what it predicted you would do.

However this machine is a jerk. Last night it predicted what you would do, and snuck into the room with a bundle of cash. If the machine predicted you will open both boxes, it put no money in either box. If it predicted you would open only one box, however, it put $5000 in one box and $5000 in the other. You get to the room and must decide to open one box or two before seeing in either.

Well if you open one box and the machine predicted it, you will win some money. But, then again, if you are going to just open one box, then the perfect predictor already predicted it so there is going to be more money in the other box too. So you should just open both boxes because the money is already there. So if you are going to open one, it is rational to really open two.

Then again, the machine is a perfect predictor, and if it knew you would do this, you should only open one box. But then …