in Brazil

I arrived in Sao Paulo on Friday, four hours late after a traumatic flight with American Airlines. Like an experiment in 'how crap can we be and still survive in a capitalist economy?' We were stuck for three hours on the runway at JFK and nobody told us what was happening. The food, which used to be halfway decent, is now just "chicken or beef?" and vegetarian options are not possible. I assume these guys are zombies, waiting to go bust under pressure from cheapo airline competition and the weight of their debts, pensions & salaries.

Brazil is awesome. The food is delicious, people are friendly, my Portuguese is terrible. Being a political obsessive I am asking everybody about Lula. The outside world is pretty favourable to him, seeing him as one of the "good Left" as opposed to the old Chavez-style populists. Everyone here, by contrast, seems very disappointed. The corruption scandals were really serious - vote buying, links to crime etc. On the other hand, there are no very inspiring alternatives and I have not yet heard any really cogent arguments against the PT's policies. The problem if anything is what they are not doing (fixing the police and education). People believe Lula will win again and it's hard to disagree.

On Saturday morning, Norman Gall from the Fernand Braudel Institute in Sao Paulo invited me to one of their reading circles. A dozen teenagers from state schools were reading Adam Smith in Portuguese. It was inspiring to see so much talent, dedication and enthusiasm. People here become very gloomy when you ask them about politics or society. I see a lot of positives. There's tremendous energy in Sao Paulo. Little charismatic churches have popped up all over the place, scandalizing educated humanists with their unsophisticated approach (anyone can become a priest without even having studied theology!) When you stop at traffic lights, a kid will jump out in front of your car and juggle tennis balls, more or less expertly. English schools are everywhere.

BTW, for a good read on Brazil today check out Norman's paper Lula and Mephistopheles. Starts from the corruption scandals and broadens out to diagnose the whole society.

Essex vaut un blog

In between very dull data importing, I thought I could find time to mention Essex. I am about to leave and spend a year at Northwestern.

I've been at two universities in my life, and have had an incredibly happy time at both of them.
Among the great things this place has are psycho ducks and rabbits, Wivenhoe (aka Wiv, The Hoe, Thundercatshoe to various deranged residents), the Wivenhoe trail, and the main building , which is cunningly designed as an ornamental maze. Then there's the amazingly international student body: a sample of young, determined and smart people from all over the world.

The best thing though is the academic community, which is closeknit, friendly, unpretentious and professional. People here are incredibly generous with their time and knowledge. The seminars are great - informal but tough. The PhD students are a very varied and interesting bunch. And Essex is imbued with the spirit of the 60s, from the not-beautiful-but-just-about-lovable architecture to the democratic optimism.

Things I won't miss? The food. Not that it's bad.... I just feel that I have eaten enough Cafe Vert paninis for one lifetime.

See y'all in a year. I'm off to Chicago!

"Are you ready for the city? Is the city ready for you?"
Futureheads The City is Here for You to Use

interpolating data in R

Here's a little function that does linear interpolation of data. For example, if you have a matrix like

[,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,] NA 1 NA NA 6 NA NA
[2,] NA 8 NA NA 5 NA NA
[3,] NA 1 NA NA 4 NA NA
[4,] NA 7 NA NA 7 NA NA
[5,] NA 9 NA NA 1 NA NA

it will be turned into

[,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,] 1 1 2.666667 4.333333 6 6 6
[2,] 8 8 7.000000 6.000000 5 5 5
[3,] 1 1 2.000000 3.000000 4 4 4
[4,] 7 7 7.000000 7.000000 7 7 7
[5,] 9 9 6.333333 3.666667 1 1 1

Columns 1, 6 and 7 have just been copied from columns 2 and 5 - i.e. this doesn't do extrapolation. Columns 3 and 4 have been interpolated linearly.

The columns of the input matrix must be either all valid, or all NA. Otherwise it will get confused. Fixes welcome.



do.interpolate <- function (m) {
if (any(is.na(m[,1]))) m[] <- rep(m[,ncol(m)], ncol(m))
else if (any(is.na(m[,ncol(m)]))) m[] <- rep(m[,1], ncol(m))
else {
p <- ncol(m)-1
m <- as.matrix(m[,c(1, ncol(m))]) %*% matrix(c(p:0/p, 0:p/p), nrow=2,
byrow=T)
}

return (m)
}

# m is a matrix with missing data
mass.interpolate <- function (m) {
for (c in 1:ncol(m))
if (any(is.na(m[,c])) & ! all(is.na(m[,c])))
stop("column ", colnames(m)[c], " of matrix m is incomplete")
valid <- c(1, which(apply(m, 2, function (x) ! any(is.na(x)))))
rvalid <- c(valid[-1], ncol(m))

for (i in 1:length(valid)) {
if (valid[i] == rvalid[i]) next # if left or rightmost cols are valid
m[ ,valid[i]:rvalid[i] ] <- do.interpolate(m[ ,valid[i]:rvalid[i],
drop=F])
}

return(m)
}

Usage: mass.interpolate(my.matrix)