The demand for bad social science

Why did nobody recognize that Lacour's research was quite implausible? One factor: people wanted to believe it. It told a good story.

Nobody much cares how, say, a Barium atom behaves. But we care very much how humans do. This may be one of social science's big problems. Theories of society, from primitive myths up to the latest carefully-identified natural experiment, are asked to play multiple roles, only some of which involve factual correctness.

Politically, governments want social science to support their narratives. Lysenko got promoted by the Soviets for telling them what they wanted to hear about evolution (well, that's biology, but the Soviets didn't do things by halves). Right now, fear of Islamists means there is a demand for studies of "radicalization". Maybe radicalization is a good object of study – maybe the concept cuts reality at the joints. More likely, it is a concept du jour which adds nothing to the idea of persuasion.

Socially, we all have a very sensible interest in encouraging the right kind of stories to circulate about the world we are living in. We want a society where people are treated equally. So, we want social science that says people are equal. The Romans wanted it the other way – stories about the essential inferiority of slaves, say. I prefer the first kind of society. But neither motivation has anything to do with the search for scientific truth.

Psychologically, all of us as thinking adults are heavily invested in our social and political beliefs. We ignore the evidence that goes against them and give extra weight to what supports them. Not surprisingly, popular science books can sell well by catering to this trend.

Social science theories often seem to coincide with changing social trends. The 70s was the heyday of "self-esteem" as a way to improve kids' performance. Now self-esteem is out and it is all about character. I would love to believe that the changes are driven by new data and theoretical advance. It seems more likely that the reverse is true. When society changes, it creates a demand for the science to support it.

I do not know how to stop this at a social level. For the individual social scientist, here is a practice worth trying. Imagine, on the way in to work every morning, that you have signed a deal with the Devil. You will insult the Gods, spit on the Lares and Penates, and mock the accepted beliefs of your society. In exchange, the Devil may grant you the power to do some decent scientific work.

Ferguson and incentive-based policy

From the Ferguson police department report:

Ferguson’s law enforcement practices are shaped by the City’s focus on revenue rather than by public safety needs. This emphasis on revenue has compromised the institutional character of Ferguson’s police department, contributing to a pattern of unconstitutional policing, and has also shaped its municipal court, leading to procedures that raise due process concerns and inflict unnecessary harm on members of the Ferguson community...

The City budgets for sizeable increases in municipal fines and fees each year, exhorts police and court staff to deliver those revenue increases, and closely monitors whether those increases are achieved. City officials routinely urge Chief Jackson to generate more revenue through enforcement....

Officer evaluations and promotions depend to an inordinate degree on “productivity,” meaning the number of citations issued. Partly as a consequence of City and FPD priorities, many officers appear to see some residents, especially those who live in Ferguson’s predominantly African-American neighborhoods, less as constituents to be protected than as potential offenders and sources of revenue.

Politicians and civil servants, please think about this...

Labour ground war problems

On excluding people from science

Following up his article on mathiness, Paul Romer argues in a blog post that:

1. Science has its own set of norms.
2. These norms are in danger from people with different norms, such as those of politics.
3. We need to exclude people with the norms of politics from scientific debate.

Points 1 and 2 are right. "Norms of politics" is not a good phrase to describe argumentative bad faith: political debate at its best is much more enlightening than that, and 'anti-politics' populism ought not to be indulged. But science does depend on norms, which must always be defended – particularly when a lot of money is being thrown at the discipline, creating incentives for shoddy work.

From a psychologically informed perspective, point 3 seems very risky. Humans are prone to many self-serving biases, which lead us to discount evidence against our views. If we truly cannot assume that reasonable scientists may differ, then we are already in the bad equilibrium. Until we are sure of that, then it is better to try hard to consider the other side's point of view, and, as Oliver Cromwell once begged of some argumentative theological scholars, "to think it possible you may be mistaken."

Professor Romer's targets in the mathiness article are people with whom has substantive disagreements. (He fairly admits that this may be bias on his part.) His blog post also caricatures Milton Friedman's famous article on scientific methodology, which has certainly been criticized before, but which is still worth reading and thinking hard about. I am not persuaded these are good targets for exclusion and shunning.

Scientific norms, any norms, do not usually fall apart because bad guys sneak in and undermine the honest people. The process is more insidious. We are all subject to pressure to publish, and the desire for fame and status; and most standards involve judgment, and grey areas. (Example: if we run many regressions and only report some of them, we risk biasing our p-values towards significance. But this does not mean we should always report every single regression we ran on some data.) These grey areas create "moral wiggle room" for us to weaken our own standards. We should all take care to avoid that, but conversely, very few of us can afford to be smug about our standards, because the pressures we face are so universal.

Frauds and cheats should certainly be excluded from science. But "mathiness" seems to be more of a grey area. All theoretical papers use simplification; all of them, I suspect, have some degree of imprecision in the map from real world concepts to parts of the formal theory. (And imprecise theories are not always bad, but that is another story.) If we shun people on this basis, we risk preferentially shunning our intellectual opponents. This itself can turn science into a dialogue of the deaf, of communities who only cite each others' papers, not because they are "cults" but because it is easier for all of us to stay in our intellectual comfort zones and receive the praise of those whom we disagree with (in my experience, every scientific community believes six impossible things before breakfast).

The best place to start is by ruthlessly excluding mathiness and other dubious practices from our own work. Computer programmers say: "Be conservative in what you do, be liberal in what you accept from others."

Linkage

Branko Milanovic on expecting institutions to do everything for us.

I have no answer to the “outsourcing” of morality. I see it as inevitable although I cannot say that I enjoy that prospect. Being an economist, I cannot even see exactly why I should not enjoy it, but I still do not.

Fish have advanced social behaviours. Work like this makes me wonder if evolutionary explanations of human behaviour should focus so heavily on the pleistocene. Maybe some basic patterns were laid down much much earlier.

Saturday poetry

If I had to tell what the world is for me

I would take a hamster or a hedgehog or a mole

and place him in a theater seat one evening

and, bringing my ear close to his humid snout,

would listen to what he says about the spotlights,

sounds of the music, and movements of the dance. 

Czesław Miłosz, "Throughout our lands"

Linkage

More from Paul Romer on science and mathiness.

As part of this process of defending science, exclusion by shunning, plays an essential role.

And:

This description of the reputational equilibrium in science shows that it is inherently unstable in the face of entry by people with the norms of politics. If a small group of people with the norms of politics can establish themselves inside a science, they can destroy the equilibrium that sustains trust and communication.

and, very pessimistically:

If I am right, that there are many economists whose judgment we should not trust and who collude with each other, how do we identify the people we can trust to make a judgment about who is trustworthy?

Governments right now are throwing money at science. Professor Romer's arguments suggest a problem with this. Oops, should I be saying that?

Grimly hilarious reading about the humanities (see earlier)

The Harvard report even admits that today, the humanities “serve only the critical function of unmasking the operations of power in language largely impenetrable to a wider public...”

Linkage

A Bangladeshi blogger standing up for freedom of speech:

But, is it really permitted to kill someone for their writing? Leaving aside freedom of speech, what does Islam itself tell us about it? I think a story from Prophet Muhammad’s life should be a reminder to the faithful. During his cumbersome days in ancient Arabia, when the prophet first began to preach Islam, he was being tortured and laughed at. One day, an old woman spread thorns all over the way he would travel. The prophet on his way found it very difficult to keep walking and his feet were hurt. But he didn’t utter a word against the old lady, who vehemently opposed Muhammad’s preaching of Islam. The story goes, this continued for days and the prophet never did any harm to the old woman. One day, he saw that the path he would walk on was clear; there was no sign of thorns. Instantly, he rushed to the old lady’s house and found her sick in bed. There was no one to take care of her. The prophet could have killed the woman instantly to avenge all the miseries imposed upon him. But he didn’t do so. He took care of the woman like his own mother and made her stand on her feet again.

 Schools that ban mobile phones get better academic results.

Linkage

Paul Romer on "mathiness" in economics.

UK government allows GCHQ to hack your computer.

A box of a trillion souls

I read Stephen Wolfram on the future of humanity:

I often imagine some kind of “box of a trillion souls” that’s sort of the ultimate repository for our civilization. Now at first we might think, “Wow, that’s going to be such an impressive thing, with all that intelligence and consciousness and knowledge and so on inside.” But I’m afraid I don’t think so.

You see, one of the things that emerged from my basic science is what I call the Principle of Computational Equivalence, which says in effect that beyond some low threshold, all systems are equivalent in the sophistication of the computations they do. So that means that there’s not going to be anything abstractly spectacular about the box of a trillion souls. It’ll just be doing a computation at the same level as lots of systems in the universe.

I read Mohamedou Ould Slahi's Guantanamo diary:

The cell – better, the box – was cooled down to the point that I was shaking most of the time.... I don't remember sleeping one night quietly....

When I failed to give him the answer he wanted to hear, he made me stand up, with my back bent because my hands were shackled to my feet and waist and locked to the floor. █████████ turned the temperature control all the way down, and made sure that the guards maintained me in that situation until he decided otherwise.

I am not sure if Mohamedou Slahi's box could in future be used for a trillion souls. Perhaps it would be too small.... I wonder, how big a box will be needed, to be the ultimate repository of our civilization?

Rules for writing software to run experiments

Most economic lab experiments are programmed in zTree, a language/server designed for the purpose by the formidable Urs Fischbacher. But zTree requires a special client, so it can't run over the web, and in other ways it is showing its age a bit. As a result, many people are trying to write new software for running experiments, both in the lab and on the web.

This list of rules comes from my experience writing betr, the behavioural economics toolkit for R. If you know R and want to try betr out, there is an introduction here. Some of these rules are things I got right... others I have learned by bitter experience.

Allow your users to write experiments in a natural idiom

Here is how I think about an ultimatum game.

1      When all subjects are ready in front of their computers,
2      the instructions are read out.
3      Then subjects are divided randomly into groups of 2.
4      In each group:
5          One person is randomly chosen to be the proposer. The other is the responder.
6          The proposer chooses an offer x between 0 and 10 pounds.
7          The responder sees this offer x and accepts or rejects it.
8      When all groups are done, profit is calculated (0 if offer was rejected; otherwise responder x, proposer 10 minus x)
9      All subjects are shown their profit.
10     After a questionnaire and payment, subjects leave.

This is a clear sequence of instructions. Some are performed in parallel for different groups of subjects. There are places where people have to wait (When all subjects are ready... When all groups are done...).

When I program an ultimatum game, I want to write something like this. If you ask me to write a set of web pages, or create an experiment by object-oriented inheritance (you wot?) then it will be hard for me to understand how my program relates to my design.

This is the most important point. If you haven't done this, you haven't helped me – after all, given enough expertise I could write my program in any language out there. Make it easy for the experimenter!

zTree gets this right ✔: experiments are sets of stages which run programs before displaying some forms to the user. betr gets it fairly right too, I think ✔.

Make the server persistent. Don't use a scripting language that exits after each request

Lots of people think "Oh, an experiment is just a website! I'll write it in PHP." This is a big mistake.

An experiment is not a collection of web pages which subjects can visit at will. It is a defined sequence of events, controlled by the experimenter, which the subjects must go through.

A typical setup for a website is: a web server such as Apache accepts HTTP requests from the clients. It responds to each request by starting a script, which gets info from the client - e.g. a form submission - passes some HTML back to the server, and exits. The server passes the HTML back to the client.

Doing things this way is fine for most websites, but it will cause you two problems.

First, all of your files will start like this (pseudocode):

// PAGE TWO
// ok where are we?
s = get_session_from_cookie()
// have we done page 1 yet?
if (s.page_one_complete == FALSE) redirect_to("pageone")
// great! let's go... wait, what if they've used the back button from page 3?
if (s.page_three_complete == TRUE) {
  error = "You mustn't use the back button, you cheater!"  
  redirect_to("pagethree")
}
// ready now... wait, what if the other subject hasn't finished?
other_subject = s.other_subject()
if (other_subject.page_one_complete == FALSE) {
    come_back_to = "pagetwo"
    redirect_to("waiting_page")
}

Et cetera. When you have many stages in your experiment, this rapidly becomes an unreadable mess.

This problem can be mitigated by using a nice library. Unfortunately, the next problem is much nastier. Here's some pseudocode for a Dutch auction. The price goes down and the first person to click "buy" gets the object.

price = starting_price - (now() - start_time) * step
if (user_clicked_buy()) {
    if (object_bought == FALSE) {
        object_bought = TRUE;
        user.profit = object_value - price;
    }
}

Looks fine?

HOOONK. Race condition.

A race condition happens when two different bits of code execute simultaneously. Sometimes two users will click buy at almost the same time. These scripts will then execute in parallel. The first one will get to object_bought and check that it is FALSE. hortly afterwards, the second one will get to object_bought and it will still be FALSE. Then, the first script will buy the object, setting object_bought to TRUE, but too late, the second script has bought it as well!

Now you have two users who bought the object; the rest of your code, which assumes only one user has bought the object, is broken; you're going to have to pay twice; and you've misled your participants – bye bye, publication.

The good news: this will happen only rarely. The bad news: rarely enough that you never catch it in testing.

You want a single server process that decides where your subjects are and what they can see, and that is running from the start to the end of the session.

zTree and betr both get this right. ✔

Use the web browser as a client

I think this is a no-brainer. If you are running online experiments, it's the obvious choice – every device has a web browser. Even if you are in the lab, you can still run web browsers on the subject computers.

Web browsers display HTML. HTML is incredibly powerful and versatile. Video chat. UI toolkits. Angry Birds. You get all these possibilities for free, often in easy-to-use components that can be copied and pasted. You can even build HTML pages with a GUI.

Most experiments are simple: some instructions, an input form, a result. But sometimes experimenters want to do more, and if you are designing for participants recruited from the web, you want your UI to be as easy as possible because, unlike in the lab, you do not have a captive audience. So, make the power of HTML available. Of course, it should also be easy to whip up a quick questionnaire or form without knowing HTML.

betr ✔ zTree ✘.

Don't use HTTP for communication

Web browsers display pages using HTML, the HyperText Markup Language. They communicate with web servers using HTTP, the HyperText Transfer Protocol. You should not do the same, however!

The reason is: HTTP is driven by requests from the client, to which the server responds. But often, experiments need to be pushed forward from the server. For example, think of a public goods game with a group size of four. Until there are four people, subjects see a waiting page. When the fourth subject joins, you want all four subjects move forward into the first round. The server must push clients 1, 2 and 3 forward.

If you try to do this with HTTP, you will have some sort of polling where clients regularly send requests to the server, which responds "keep waiting" until it is ready. This is a horrible kludge. You are reliant on your clients to keep polling (but maybe some guy closed the browser window and you don't know that!) You have to manually work out when everyone has moved on, which means keeping track of state. Argh.

Another example: suppose you have a market. The server takes bids and offers and calculates prices. Whenever the price changes, you want all clients to update. With http, this is going to be impossible.

Luckily, modern browsers have a new technology called websockets which allow for two way communication between client and server. This is probably the way to go. Load a basic HTML page which connects to your server using a websocket. Then send new HTML pages and other updates via the websocket.

zTree obviously gets this right by not using http ✔ . Sadly, betr gets it wrong ✘. Oliver Kirchcamp pointed this out to me. I'm thinking about how to fix it in future versions.

Make testing and debugging easy

Testing and debugging zTree is not much fun. It involves manually opening one window for each client, and running through the experiment. Then doing it again. And again. And again.

But testing is really important. If you screw up in a real session, you instantly waste maybe £500 in experimental payouts. Worse still, if your experiment didn't do what you said it was going to, your subjects may not believe experimental instructions in future.

And, you typically need to test a lot, because you want to test distributions of outcomes. For example, suppose in your design, subjects are randomly rematched. You want to make sure that subjects are never rematched with the same partners. This requires testing not once, but many times!

Failure to test leads to problems. Here is a zTree code snippet that is around on the internet, e.g. here. It lets you pick a random number uniformly between 1 and max.

rand = round(random()*(max-1), 1) + 1

There is just one problem: this snippet is wrong. It does not create a uniform distribution. 1 and max will be half as likely as the numbers in between. If experimenters test that their random number is really uniformly distributed, they will catch this bug. Otherwise, they will have bad code, and perhaps mistaken results.

A good way to test is to create "robots" that can play your experiment automatically. This requires some clever design. Nice features would include

• printing out the HTML pages sent to subject
• being able to mix robots and real users (so you can change one users' actions while keeping others the same)
• create robots easily from records of previous play

All of this should be provided by good modern experiment software.

betr gets a partial ✔. There is a nice replay() function which can do a lot of this stuff. zTree ✘.

Experiment sessions should be idempotent

This is related to the previous point. It also helps for crash recovery. Idempotency means that if it gets the same inputs, an experiment should produce the same results. Wait, don't we want to do a lot of randomization? Well, yes, but we also want to be able to recover from crashes, and to replay experiments exactly, by replaying commands from the clients. Among other things, this means that everything your clients do should be stored on disk so you can replay it. You should also take care to store the seeds used for random numbers.

Both betr ✔ and zTree ✔ encourage this. zTree gets a bigger tick because (I believe) it enforces it.

Let experiments be written in an existing language, don't create your own

This is the same principle as "use the web browser as a client". It's about giving your experimenters tools. zTree can do a lot, but you can't (for example) define your own functions. So, if you want to repeat a bit of code... you have to copy and paste it. A good general purpose language gives the user access to many many libraries that do useful things.

Here betr ✔ wins against zTree ✘. On the face of it, R is a strange choice of language to write a web platform in! But it has a lot of power, and many academics use it already. Python would be another good choice. (PHP would not.)
 

It must be easy to share data with clients

An awesome feature of zTree is that if you write a piece of user interface code which uses a variable X, and X changes on the server, that is automatically reflected on all clients' screens. Doing this is hard, but very useful: think about displaying prices in a market experiment.

zTree ✔ betr ✘.

There are many alternatives to zTree out there. (Some of the most interesting: oTree, moblab, sophie, boxs). I expect that soon, some of them will start to get traction and be used more widely. I look forward to a wider choice of powerful, easy software to run experiments!

Linkage

Coolest protest masks. It's the Kale Bloc!

Do nice people or nasty people get ahead, in our economic system? This is a vague and complicated question but not a meaningless one. Here is a data point.

Rebecca Roache beautifully illustrates my point about group identity in politics.

Michael Clemens argues that if the EU opened its borders completely, the sky would not fall in:

Gallup pollsters go to pretty much all countries, and in each country, they ask around 1,000 adults the same set of questions every year. One of the questions is about whether or not they would like to emigrate, and if so, where.
The bottom line is that Europe's overall population would rise by 10 percent if everyone who told Gallup they would like to move to Europe could do so. Germany's population would rise by 23 percent, because it is a particularly prosperous and desirable destination.
This is the best direct evidence we have. I say this evidence is only suggestive, because we don't know the extent to which stated desires reflect real behavior.

Indeed. My suspicion is that this is not very informative. It's like when people threaten to leave the UK after the Tories get in; some will, most won't. I would find it more interesting to look at historical examples of countries with open borders (actually almost everywhere, pre-1914: the real age of mass migration was in the 19th century). In any case, this is a great read:

In the United Kingdom, asylum seekers are net takers of benefits because they are banned from working. That is, UK voters apparently support policies that force asylum seekers not to generate tax revenue. Then some of the same voters complain about asylum seekers because they do not generate tax revenue.

Censor these dreadful arguments

Nice quote from a long article about political correctness by Nick Cohen.

• A belief that a misplaced word or awkward phrase reveals your opponent’s hidden meanings and unquestioned assumptions. The wised-up need only decode the slips, and the mask will be ripped off and everyone will see the oppressor’s true face.

Yes, this is a terrible form of argumentation which needs to die. It is a lazy substitute for actually engaging with one's opponent's views.

Electoral reform?

I joined the Electoral Reform Society yesterday, because... well, it's obvious why. First Past The Post was really unfair in this election: I think both UKIP and the Greens are nutters, but the people who voted for them deserve representation. Thatcherites like me, who believe in free entry into markets, should not support a political system which creates a duopoly. And most importantly, FPTP is now encouraging the breakup of the UK, a country I rather like.

What are our chances? There is certainly a huge wave of support, and it is not just from the usual suspects – perhaps ordinary people are coming to see that this is more than an obscure technical issue. On the other hand, we had a referendum on changing the voting system just 4 years ago. And since the Tories – alas – have an absolute majority there is no reason for them to do it again.

The question is whether Labour will now finally get behind electoral reform. If so, then we could easily get reform in 2020. Labour have traditionally benefited from the two-party system, but they have had their Scottish ground cut out from beneath them, and this will not change easily. If you are in Scotland and left wing, will you vote for a centrist party – or a more Left party that promises to support a progressive coalition? Labour got attacked for being too Left in England and being "red Tories" in Scotland. They can decentralize their party structure but this will only help so much. So, perhaps they will see the sense in getting the benefit of the many Scottish voters who supported them, and who are now represented by just one MP.

The country's establishment should also get on board. The great and good of the UK surely do not want to watch while their country fragments. Electoral reform cannot stop nationalism, but it does prevent a single party with a regional support base – and that describes all our parties now – from dominating the whole of politics. For that reason it must be centripetal.

Politics and hatred

 With my social scientist hat on, I study group identity – how people behave differently to ingroups and outgroups. My experimental subjects are mostly students. If one divides them by sex or ethnicity, they are quite unlikely to be mean to the other group. This is not so surprising: most young people strongly believe that sexism and racism are wrong. But, dividing them by politics works much better. Leftwingers are happy to discriminate against rightwingers, and (maybe?) vice versa. (I, Denise Laroze and Arndt Leininger have some work using this approach: here is a presentation.)

This is not just true in the lab. Someone once refused to shake my hand because I was a Conservative. Today twitter is ablaze – when is twitter not ablaze? – with photos of the anti-Tory protest in London.

My impression is that political hatred tends to be stronger from the Left to the Right than the other way round. Perhaps this is because many on the Left are idealists with a strong concern for social justice and compassion. To be against those things, surely you must be evil!

But of course across history there have been many Right wingers who evince hatred for their opponents. Political scientists talk about "politicide": destroying a political movement by mass murder. After Suharto gained power in Indonesia, up to a million people were murdered for being Communists.

Thinking about politics through the lens of group identity can be analytically helpful. Why do people vote when notoriously, a single vote will not change the result? Well, people don't go to football matches because they have calculated that the sound of their cheering will cause their team to win, but because they enjoy supporting their team. Maybe voting is like that.

It would also be a good idea if politically active people were more self-aware about this "group identity" aspect of their beliefs. Be passionately committed to your political ideals – God knows, we do not need more apathy. But understand that this engagement works partly on a primitive, non-cognitive level of your brain. If we all knew this, democratic politics might be a bit less rancorous.

Meanwhile, when a left wing person tells me I am evil, I have an easy response: "I don't think you're evil!

"I just think you're stupid." :-D