The blog of John D. Cook

I recently found out about a book that was published earlier this year, The Cult of Statistical Significance by Stephen Ziliak and Deidra McCloskey. The subtitle is sure to stir up controversy: How the Standard Error Costs Us Jobs, Justice, and Lives.

From the parts I’ve read it sounds like the central criticism of the book is that statistical significance is not necessarily scientific significance. Statistical significance questions whether an effect exists and is unconcerned with the size or importance of the effect.

Significance testing errs in two directions. First, in practice many people believe that any hypothesis with a p-value less than 0.05 is very likely true and important, though often such hypotheses are untrue and unimportant. Second, many act as if a hypothesis with a p-value greater than 0.05 is “insignificant” regardless of context. Not only is the 0.05 cutoff arbitrary, it is quite common to say there is evidence if p = 0.049 and to say there is no evidence if p = 0.051. Common sense tells you that if 0.049 provides evidence then 0.051 provides slightly less evidence rather than no evidence.

The book gives the example of Merck saying there is “no evidence” that Vioxx has a higher probability of causing heart attacks than naproxen because their study did not achieve the magical 0.05 significance level. The book argues that “significance” should depend on context. When the stakes are higher, such as people suffering heart attacks, it should take less evidence before we declare an effect significant. Also, if you don’t want to find significance, you can always reduce the size of your study to decrease your chances of finding significance. [I have not followed the Vioxx case and have no opinion on its specifics.] In addition to the Vioxx case, Ziliak and McCloskey provide case studies in economics, psychology, and medicine.

Whenever someone raises objections to significance testing the reaction is always “Yes, everyone knows that.” Everyone agrees that the 0.05 cutoff is arbitrary, everyone agrees that effect sizes matter, etc. And yet nearly everyone continues to play the p < 0.05 game.

Related posts:
Origin of “statistically significant”
Five criticisms of significance testing

Andy Hunt posted an article this morning entitled Science Failure and Cubicle Brain Death. He explains that one reason it took so long to discover that adult animals could grow new brain cells was that such growth doesn’t happen in laboratory conditions. To grow new brain cells, animals need stimulation that a sterile lab environment does not provide. People need stimulating environments too. Little things matter.

… things like the pen and paper you use, the decorations at your desk, the lighting and ceiling height of your cubicle all have a measurable effect on your cognitive processes.

Joel Spolsky talked about this in the latest StackOverflow podcast. His company often faces criticism for spending so much money on office space for developers. But as he put it, the difference between depressing and stimulating office space may amount to whether you devote 4% or 6% of your total budget to rent. The extra investment in office space allows you to recruit more competitively for top talent and makes the people you hire more productive.

Related posts:

Selective use of technology
Brain plasticity
Getting to the bottom of things

Here’s something that looks like it could be handy: The Matrix Cookbook.

(I just realized that not everyone know the context of the link posted above. The Matrix Cookbook is a summary of facts about mathematical matricies. Feel free to leave your own joke about sci fi movies and food preparation in the comments.)

Michael Feathers wrote one of my favorite books on unit testing: Working Effectively with Legacy Code. Some books on unit testing just give abstract platitudes. Feather’s book wrestles with the hard, messy problem of retrofitting unit tests to existing code.

The .NET Rocks podcast had an interview with Michael Feathers recently. The whole interview is worth listening to, but here I’ll just recap a couple things he said about refactoring that I thought were insightful. First, most people agree that you need to have unit tests in place before you can do much refactoring. The unit tests give you the confidence to refactor without worrying that you’ll break something in the process and not know that you broke it. But Feathers adds that you might have to do some light refactoring before you can put the unit tests in place to allow more aggressive refactoring.

The second thing he mentioned about refactoring was the technique called “scratch refactoring.” With this approach, you refactor quickly without worrying about whether you are introducing bugs in order to see where you want to go. But then you completely throw away those changes and refactor carefully. Sometimes you need to do a dry run first to see what patterns emerge and determine where you want to go.

Both of these observations are ways to break out of a chicken-and-egg cycle, needing to refactor before you can refactor.

Mathematics Diary posted the following identity this morning. If a + b + c = π then

tan(a) + tan(b) + tan(c) = tan(a) tan(b) tan(c).

I’d never seen that before. It’s striking that the sum of three numbers would also equal their product. In fact, the only way for the product of three numbers to be equal to their sum is for the three numbers to be tangents of angles that add up to π radians. I’ll explain below why that’s true.

First, we can generalize the identity above slightly by saying it holds if a + b + c is a multiple of π. In fact, it can be shown that

tan(a) + tan(b) + tan(c) = tan(a) tan(b) tan(c)

if and only if a + b + c is a multiple of π. (Here’s a sketch of a proof.)

Now suppose x + y + z = x y z. We can find numbers a, b, and c such that x = tan(a), y = tan(b), and z = tan(c) and it follows that a + b + c must be a multiple of π. But can we chose a, b, and c so that their sum is not just a multiple of π but exactly π? Yes. If a + b + c equaled kπ for some integer k, pick a new value of c equal to the original c minus (k-1)π. This leaves the value of tan(c) unchanged since the tangent function has period π.