I Hate Your Stupid Paper: Gilens & Page

(An occasional series in which I hate on a paper and explain why.)

Gilens and Page, 'Testing Theories of American Politics: Elites, Interest Groups, and Average Citizens'

Who really governs in American politics? Is it the ordinary citizen, Joe Schmo? Or his wealthy cousin, Joseph Schmo III? Or maybe interest groups have the power, or perhaps only business-related interest groups.

To answer this question, Gilens and Page collect information about the preferences of each of these possible influencers, and use multiple regression to find which groups' preferences correlate significantly with actual policy outcomes. The results look bad for US democracy: only elite citizens and business interests matter. This garnered some publicity in the press, giving succour to student radicals everywhere: "meet the new boss, same as the old boss".

This paper is about an important topic - who really decides what happens in democracy? The data collection effort was huge. The empirical results are worth thinking about. I personally find the conclusions quite plausible. So what's the problem?

Short answer: the conclusions may be plausible, but they don't follow from the results.

Problem I: exogeneity

This is such an old chestnut/cheap shot that I feel bad bringing it up, but correlation ≠ causality. And here it really matters: the paper is news because of the interpretation that the policy system responds to elite citizens' preferences, but not to normal citizens' preferences. That is clearly a claim about causality.

But we don't know whether any of these groups' preferences are actually causing policy changes, because they might be correlated with something else that is doing the real work.

Here's a story: policy is actually made by enlightened civil servants, working hard to find what is best for the US. Citizens also form opinions. The elite citizens are better informed. As a result, their opinions and policy preferences are like those of the hard-working civil servants. Joe Schmo, by contrast, gets his opinions from Fox News. Result: policy outputs correlate with elite preferences.

Implausible, perhaps, but the story is wholly compatible with the evidence in the paper. If you don't like it, try this one. Policy is made by the Bilderberg Group, who have Obama's head on a stick and they move his mouth using puppet strings. The Illuminati own the media and shape people's opinions. But the rich are more easily influenced, because they don't get THE REAL DOPE from Fox News. So, rich citizens' opinions conform more with the policy of their Bilderberg/Illuminati masters.

You can play this game all day, and it is not hard to make these stories more realistic. Conclusion: the paper cannot show either that elite preferences do affect policy, or that ordinary people's preferences do not.

Getting real causality would require observing some exogenous change in citizen preferences - a change independent of anything else that could affect policy - and seeing how political outputs responded. This is hard if not impossible. You cannot experiment on US political opinion at any useful scale, and there are probably no "natural experiments" that change public opinion without also changing something other relevant factor.* So it is easy to excuse these limitations.

What is harder to excuse is that those limitations barely get a mention. I read through this paper waiting for the jump scare - thinking "soon they're going to show the amazing way they aim for causality! Or, at least, they will have a responsible adult discussion about this limitation of their results."

Jump scare never came. In one sentence, the authors admit: "... it is also possible that there may exist important explanatory factors outside the three theoretical traditions addressed in this analysis." Everywhere else, the language of causality is pervasive: wealthy citizen preferences have "impact" and "influence" on politics. That language is incompatible with the sentence I just quoted.

The authors ask an interesting question. But their research design cannot answer it.


* It's not even obvious that politics ought to respond to exogenous changes in citizen preferences - changes unrelated to, e.g., political reality. Suppose evil Commies put an opinion drug in the water supply, and everyone woke up wanting higher taxes: should policy change accordingly?


Problem II: bad political philosophy

Here's one concept of democracy: democracy means doing the people's will. If the majority wants nuclear disarmament, a democracy disarms. If it wants lower/higher taxes, taxes go down/up.

This "populist" interpretation of democracy is appealingly simple, but it has a serious problem.** It requires that people actually have political preferences. Unfortunately, public opinion theorists long ago noticed that most people's "opinions" about politics are like their opinions about Venusian geology: if you question them, they will obligingly give an answer, but it is likely to be ill-informed, unrelated to other ideas in their heads, and to change when you ask the same question next week.

Note that this also could explain the paper's result. If elite citizens have opinions that relate at least somewhat to reality, and ordinary citizens' opinions are just noise, then elite citizens' preferences will probably be closer to policy outcomes.

In any case, basing democracy on "opinions" like that seems foolish. Who should decide the level of healthcare spending in the US? The average voter, who has no clue about the future demand for healthcare? Or should we, the people, hire an expert? We'd better make sure we can fire the guy if he doesn't do his job; that will also give him an incentive not to screw up too badly. We could call these people... politicians.

This alternative idea still links democracy to what people want - but making their real needs count, not their political pseudo-opinions. Gilens and Page mention it but dismiss it as follows:

The “electoral reward and punishment” version of democratic control through elections—in which voters retrospectively judge how well the results of government policy have satisfied their basic interests and values... might be thought to offer a different prediction: that policy will tend to satisfy citizens’ underlying needs and values, rather than corresponding with their current policy preferences. We cannot test this prediction because we do not have—and cannot easily imagine how to obtain—good data on individuals’ deep, underlying interests or values, as opposed to their expressed policy preferences.

Notice that now, we are basing our political philosophy on data availability: "Some people say real interests matter, but that's too hard to measure! We'll use populism as our benchmark instead."*** Also, why can we not find out about individuals' deep interests? I guess most people value wealth, health, happiness and security. Does a system provide that for the elite? For the majority? If there's a conflict, what happens? These questions are as easy or easier to answer than the one the paper addresses.

So, Gilens and Page. You threw out a useful concept of democracy in favour of a populist chimera, so as to do empirics that do not work. Your paper has garnered press attention! It is influential and cited! Some say it has proved American democracy is dead! I hate your stupid paper.


** Actually, two problems. The other is that even if people have clear preferences, there may be no coherent way to aggregate them to decide "what the majority want". This is the topic of Arrow's Theorem; William Riker used it to attack the "populist concept of democracy" in Liberalism against Populism. 

*** I am being a bit unfair here. G & P consciously discuss and defend the importance of this populist conception of democracy. Read the paper and make up your own mind (a good idea in general; even "stupid" papers are usually more informative than their media coverage).