Glossy science versus messy science

TL;DR: the editor said a bad thing, and now I am sad about incentives for researchers.

 Just got a desk reject for my honesty paper. The journal editor's comments were brief, but very thoughtful and useful, and I'd much rather have a quick decision than wait 3 months for the same one.

There was one point I disagreed with. The editor commented that my two measures of honesty didn't correlate highly at country level, and this left the reader unsure of how to interpret the results.

Well, sure. Here below is honesty measured by a coin flip. (You ask people to flip a coin and report the result, and you give them a reward for getting heads. If everyone was honest you'd see 50% reporting heads; the proportion actually reporting heads gives you an estimate of the proportion of honesty in the group – in this case, each of the 15 countries.)

 And here is honesty measured by answers to quiz questions. (The questions were so hard that you could only answer, I claim, by cheating and looking up the answers on the internet. See the paper for more details.)

The two measures indeed have some big differences. For example, the Japanese sample goes from really dishonest in the coin flip to really honest in the quiz. (Why? Well, gambling is illegal in Japan. Maybe Japanese people saw the coin flip as already a bit naughty, and felt less inhibited about unethical behaviour in that context. Or maybe [insert your preferred post hoc explanation here].) So, yes, these differences make it harder to interpret the results.

But suppose I only had the coin flip measure – like, ahem, several other papers on this topic. Then my results would be really easy to interpret! Of course, they would also be potentially misleading. No single measure is guaranteed to capture exactly what we want, all have potential confounds like attitudes to gambling, or attitudes to quizzes. That's why we need to use multiple measures and check robustness of our results.

I think we should be careful about the incentives we create. If we only demand data that tells a simple clear story, that is what we will get: glossy science that sells well, instead of messy science, which is the reality.

Maybe what I should have done is published one paper with the coin flip results. Then I could publish a  second paper saying "Ah, but results are different in the quiz experiment!" Ha ha. Of course I would never....