How fast do institutional changes take effect?

Today I saw a presentation of a field experiment evaluating different ways to deliver aid. One was the standard method which had been used until that point. The other two “treatments” were new. The evaluation was over the course of a year. (There were lots of pictures of smiling villagers... lab guys don't have those.)

How long do we have to wait before new institutions settle down and we can evaluate them “in equilibrium” – meaning, not any rigorous game-theoretic concept, but just that people have somehow got used to them, and that all the changes have worked through the system? In this case I felt that a year was too little. Aid recipients are unlikely to be naïve about the fact that new institutions are being tried and evaluated, and they may therefore behave in a special way in the first season of a change.

I worry in general that social scientific evaluation is too short-termist, and that the tools of statistical analysis can encourage this. For major institutional changes, there is a good case that the smallest possible “independent” unit of observation is a generation. Until people have grown up under a new system, we are not sure that its full effects have been worked out.

In this context, advanced statistical analysis can actually be a step backward. For example, consider this paper which estimates the effect of democracy on GDP – a traditional hobby for political scientists. Now as everyone in the field knows, just looking at democracies versus dictatorships and comparing averages will not be informative, because these countries differ in many many other ways. So instead the paper looks at the few years before and after a change from dictatorship to democracy, and estimates the switchover effect from that. But this is crazy, because such massive changes to social institutions are not remotely likely to have all their effects within a few years. After all, many political decisions have ramifications that span decades – think of the choice to create the NHS, or Lloyd George's introduction of old age pensions, or Nixon's visit to China. Evaluating political institutions after seven years is like evaluating a new fitness regime after a week.

I can think of several cases where my previous beliefs were probably based on too short a run of evidence. For example, I assumed that privatization of e.g. water utilities was a good thing, because the privatized utilities performed better than public counterparts elsewhere. But a lot is going to depend on the first generation of entrepreneurs who take over, and these may not be the same as the second or third generation of entrepreneurs who inherit the system. In the 1950s, nationalization must have seemed as obvious as privatization did in the 1990s. I am not saying that privatization was a mistake – I still support it – but I am less confident of the evidence base.

My colleagues seem to be making a similar mistake about the effect of the Research Assessment Exercise (now the Research Excellence Framework) for UK academia. Everyone I know who was around in the 80s, when this came in, says that it swiftly forced a lot of unproductive, “dead wood” academics to either shape up or leave the system. So they are basically positive about it. (Well, Essex political scientists would be, wouldn't they?)

The question is whether it is still having the same effect now. When a new institution is imposed, there are two kinds of adjustment: people adjust to the institution; and the institution is adjusted to the people. After all, nobody wants to live under permanent revolution, so initially harsh conditions are gradually softened, informal routines grow up that may subvert the official rules, et cetera.

In this context it is pretty alarming to consider the Conservatives' and then New Labour's regime of targets for the NHS (known by some as “targets and terror”). Again, I have heard people in the industry talk about the salutary initial effects of having managers asking “why isn't this bed being used”? But now look where targets and terror have got us.

If this argument is right, we will often be unable to evaluate institutional changes rigorously until long after the fact – even if we are doing randomized controlled trials, which is often impossible. So, how can we decide what changes to make in the here and now? Two things might help.

First, a historical perspective won't tell us what will happen, but will at least give us a sense of what can happen. Without history we are doomed to parochialism. As Churchill said, "Study history, study history. In history lies all the secrets of statecraft."

Second, a sense of principle might often be a good guide. Actually, another Churchill quote is relevant: "In life the only wise course is to follow the course of duty and not of interest. Every man knows what his duty is. But it is not given to many to know their true interest." To apply this to academia: we may not know, now or ever, the true efficiency effects of such-and-such a government evaluation framework, or of the practice in an increasing number of European universities of – no joke – paying bonuses for top journal publications. But every researcher should feel that the search for truth is sacred, that it requires rigorous and demanding standards of honesty in the muddy waters of empirical analysis, and, therefore, that attempts to import monetary incentives, or impose pressure to publish, should be met with great suspicion.