The Cartography of Bullshit

With the gutting of foreign coverage by most U.S. newspapers and the need to populate infinite Web space with content, a new creature has emerged: the foreign affairs blogger. Max Fisher, who hosts the Washington Post’s WorldViews page, is a leading exemplar of the species. Fisher’s newsy nuggets are often low-priority zeitgeist items that may or may not be vignettes of greater themes: examples in recent days include the tunnel-smuggled delivery of KFC chicken into Gaza, the video of the Czech president possibly drunk, a staff-passenger brawl at Beijing airport, and New Zealand’s “war on cats.” Fisher also concocts FAQ-style explainers on places in the news that he judges to be obscure to his readers (Chechnya and Dagestan, Central African Republic, Mali). And he is very keen on global surveys, whose results he summarizes, augments with his own interpretation, and typically renders with color-coded maps that drive home the key message.

This week, Fisher proposed to his readers what he titled “A fascinating map of the world’s most and least racially tolerant countries.” The deep-blue, racially tolerant areas included the US, UK, Canada, Australia, Scandinavia, and much of Latin America. The deepest-red, or most racially intolerant, countries were India, Bangladesh and Jordan. Russia and China fell in the middle; much of Africa was left out for lack of data, but South Africa came out light blue (highly tolerant), and Nigeria light red (highly intolerant). Other highly tolerant countries included Pakistan and Belarus.

A cursory glance at this distribution of results would suggest something deeply suspect about the exercise; moreover, anyone who studies the concept of race knows that it is hard enough to operationalize in a single-country context, let alone in cross-national comparison. Still, Fisher soldiered on, offering bullet-point findings: “Anglo and Latin countries most tolerant,” “Wide, interesting variation across Europe,” “The Middle East not so tolerant,” and the like. He offered country-level speculation: tolerance was low in Indonesia and the Philippines “where many racial groups often jockey for influence and have complicated histories with one another,” and lower in the Dominican Republic than in other Latin countries “perhaps because of its adjacency to troubled Haiti.”

Where did these numbers come from? As Fisher explained, they came from the long-running World Values Survey, which has polled attitudes around the world for decades. Fisher was drawn to the topic by news of a new paper, by a pair of Swedish economists, on the links between economic freedom in a country and its level of tolerance. (The paper was described in a post at Foreign Policy, itself a hub of foreign-affairs blogging.) To measure racial tolerance in particular, the authors used question A124_02 in the World Values survey, which asks respondents whether they would “not like to have as neighbors people of another race.” Intrigued, Fisher went back to the survey itself and, as he put it, “compiled the original data and mapped it out in the infographic” that led his post.

Although the results don’t pass the sniff test in the first place, I took a look at the data as well, in an effort to identify the exact problems at play. It turns out that the entire exercise is a methodological disaster, with problems in the survey question premise and operationalization, its use by the Swedish economists and by Fisher, and, as an inevitable result, in Fisher’s additional interpretations. The two caveats that Fisher offered in his post – first, that survey respondents might be lying about their racial views, and second, that the survey data are from different years, depending on the country – only scratch the surface of what is basically a crime against social science perpetrated in broad daylight. They certainly weren’t enough to stop Fisher from compiling and posting his map, even though its analytic base is so weak as to render its message fraudulent.

For one thing, the values for each country are indeed from different years, some in the past decade, others as old as 1990. As Fisher put it coyly, “we’re assuming the results are static, which might not be the case.” Indeed: by a rigorous methodological standard, this would be enough to throw out the cross-country comparison in the first place.

Second, a visit to some of the other tolerance questions in the A124 series reveals absurd results and design idiosyncrasies that should render the results of question A124_02, on race, suspect. The other questions ask respondents if they would accept a neighbor who had various other traits: homosexuality, a different religion, heavy drinking, emotional instability, a criminal record, and so on.

To take an example of the weakness of the data, it would appear that in Iran in 2000, only 0.9 percent of respondents “mentioned” an objection to having a homosexual neighbor, whereas in 2007, 92.4 percent mentioned it. In Pakistan in 2001, according to the survey, 100 percent of respondents “did not mention” objection to a homosexual neighbor. These are obviously particularly buggy examples, but these are the data points that the survey offers for analysts to work from; readers can visit the database to form their own opinion.

Moreover, the menu of traits available in the survey for respondents to tolerate or not tolerate varied by country. Thus, Iranians were asked about Zoroastrians; Puerto Ricans, about Spiritists; Tanzanians, about witchdoctors; Peruvians, inexplicably, about “Jews, Arabs, Asians, gypsies, etc.” (A124_33). In other words, the question about race was presented as part of a different menu of questions depending on the country, another red flag signaling a need for caution in isolating it and using it to produce grand findings. And further issues abound: as Fisher noted, self-reporting of prejudice is unreliable to begin with; as the scholar Steve Saideman pointed out, the “neighbor” question is not the best measure of tolerance; and so on.

But the biggest problem, of course, is that “race” is impossible to operationalize in a cross-national comparison. Whereas a homosexual, or an Evangelical Christian, or a heavy drinker, or a person with a criminal record, means more or less the same thing country to country, a person being of “another race” depends on constructs that vary widely, in both nature and level of perceived importance, country to country, and indeed, person to person. In other words, out of all of the many traits of difference for which the WVS surveyed respondents’ tolerance, the Swedish economists – and Fisher, in their wake – managed to select for comparison the single most useless one.

Fisher has an active social-media presence and his posts circulate quite broadly among international-affairs geeks and journalists in many countries; this one found the usual echo on the networks, plus a fair amount of skepticism. In India and Pakistan, Twitter readers were shocked by India’s ultra-high and Pakistan’s ultra-low racial intolerance ratings, both on their own merits and in comparison to each other. Lakshmi Chaudhry and Sandip Roy, at India’s Firstpost, wrote a detailed objection. (Less productively, Philip Weiss at Mondoweiss objected that Fisher’s map excluded Israel, implying that this deliberately overlooked racism in Israel – a spurious accusation, since there are no data available for Israel for question A124_02 in the WVS in the first place.)

On Twitter, Fisher engaged with Saideman but brushed off other queries, tweeting archly: “Coincidentally, readers from red countries are much more likely to say they doubt the methodology behind this study.” When I raised many of the issues in this post, he offered no response or acknowledgment at all, except to block me on Twitter. (That’s why I’m not bothering to seek comment from him before running this piece.) He summarized a few of Saideman’s objections in a follow-up post, but much of this goes down the rabbit-hole of political-science arcana about ethnic conflict and, for some reason, the specific case of Somalia. A more intellectually honest move would have been to take down the map and explain to readers why the exercise was doomed from the start.

Instead, we are left with a shiny color-coded “fascinating map” on the Washington Post site that sends a strong message of Western, Anglo-Saxon moral superiority, assorted with a mystifying portrayal of the rest of the world, and accompanied by near-gibberish interpretations – all based on a methodological process that fails pretty much every standard of social-science design and data hygiene. In other words, pseudo-analysis that ends up, whether by design or by accident, playing into an ideological agenda.

But the problem here isn’t the “finding” that the Anglo-Saxon West is more tolerant. The problem is the pseudo-analysis. The specialty of foreign-affairs blogging is explaining to a supposedly uninformed public the complexities of the outside world. Because blogging isn’t reporting, nor is it subject to much editing (let alone peer review), posts like Fisher’s are particularly vulnerable to their author’s blind spots and risk endogenizing, instead of detecting and flushing out, the bullshit in their source material. What is presented as education is very likely to turn out, in reality, obfuscation.

This is an endemic problem across the massive middlebrow “Ideas” industry that has overwhelmed the Internet, taking over from more expensive activities like research and reporting. In that respect, Fisher’s work is a symptom, not a cause. But in his position as a much-read commentator at the Washington Post, claiming to decipher world events through authoritative-looking tools like maps and explainers (his vacuous Central African Republic explainer was a classic of non-information verging on false information, but that’s a discussion for another time), he contributes more than his weight to the making of the conventional wisdom. As such, it would be welcome and useful if he held himself to a high standard of analysis – or at least, social-science basics. Failing that, he’s just another charlatan peddling gee-whiz insights to a readership that’s not as dumb as he thinks.

80 thoughts on “The Cartography of Bullshit

  1. Siddhartha, thanks for digging into this. The WP posted a correction about the Bangladesh data. Apparently somebody at WVS (doh!) switched the “yes” and “no” answers, so we’re actually only 28.3% intolerant. Who knows what other f-ups they made in this run-like-a-tight-ship study.

    • hehehe… this is hilarious! this survey and the subsequent analyses are a perfect case of garbage-in-garbage-out.

  2. People for blue countries are very politically correct. they don;t admit their racial bias and hence “blue”. The most tolerant..freely admit they rather live among people of similar background but because of their openness their are usually more tolerant. This study perhaps must be used in reveres. Blue means red means blue.

  3. Yeah, South Africa being blue should be a very red flag to anyone who knows that apartheid didn’t just up and vamoose in ’94. The attitudes and racism are still about.

  4. Thanks for your discussion and clarification of the, as you say, methodological problems of this information. As a generic white man, it initially pleased me to see westerners portrayed as more socially forward and tolerant, but with your analysis, you have simply illustrated quite well how we all need to take data presented as fact with a large soupçon of salt.

    I’ll be more wary in the future. Good job!

  5. I would also like to add as a caveat that the author’s assertion, “whereas a homosexual, or an Evangelical Christian, or a heavy drinker, or a person with a criminal record, means more or less the same thing country to country” is in fact not the case at least with respect to sexuality. Queer Studies (as well as Women and Gender Studies) has certainly revealed that “Homosexuality” (however that may be defined) does not necessarily mean the same thing from country to country and is as much a socially constructed identity as race. One could also argue that the notion of what constitutes heavy drinking varies greatly from place to place given that social meanings attached to drinking vary widely from society to society.

  6. Humph, what a load of bumph! South Africa, for one, should be bright, bright red. As Mr Sullivan points out, the racism of apartheid is still widely spread, but, what he omitted to say, was that the present government has implemented racially biased laws as did the apartheid government. Even Zuma, the pathetic president of South Africa, has publically stated that minorities, read whites, have less rights than the majority solely based on the fact that they are in the minority. Max FIsher really needs to get a real job!

  7. I saw the chart recently and suspected inconsistencies, misrepresentations, intentional ommissions and data manipulation from the start. Thank you so much for this excellent analysis and rebuttal.

  8. I’d like to hear form Max Fisher on how far he thinks his map is an accurate reflection of his international experience or was it just a smart numbers exercise he came up with in order to have something new to say and now he feels he has to stick to it.

  9. Actually, Fisher does not claim to use any new methodology or survey. He only claims to have “compiled the original data and mapped it out on the infographic”. He is quiet frank in mentioning the caveats of the study. However, it is clearly unacceptable to Indians that they are painted as 2nd most racist countries. Article written by Siddartha Mittar is particularly schizophrenic as he demonizes Fisher for blogging by using the same medium of blogging, criticizes Fisher for being unscientific without any data to support his own claims. It’s amusing that we accept the words of quacks, fraudsters, and religious books/gurus as the word of God with zero evidence; but can’t stomach the scientific data that, even though imperfect, gives a reasonable window into these attitudes.

    Although it is possible, I doubt that Fisher hates India and loves Pakistan. Rather than shooting the messenger, we will do better by improving the tolerance and promoting the inherent equality of all people. However, knowing that old habits die hard, at least we can produce better scientific reasoning or data to argue against the article. Even attempting to study that in an open & honest fashion may bring positive change. Alas, any study by Indians in this field is likely to be cooked. And if miraculously it is not, it will be drowned by cacophony & name calling.

  10. Piyush : we are missing the point. Wrong data & wrong conclusions are worse than no data. What Max Fisher has concluded is absolutely unscientific by any measure. Just because it comes from a survey does not make it any credible. And just because Max mentions a caveat does not absolve him of the crime of misinterpretation. And the caveat he mentions is not even the full truth. So, i have no clue what makes you believe that Max is right in making sweeping conclusions. This has nothing to do with me being an Indian or Indonesian or anything…….

  11. I agree with this article about India. Racism is everywhere in India. Especially if you go towards semi-urban and rural India. The fact that India has highest amount of reservation is evident enough of inequality.
    The problem is not that there is a problem. The problem is that India does not accept that there is a problem.

    • Gautham : There is a diference between race and caste. The fact of the matter is most Indians, especially those you mention – those rural areas – would not even know what is meant by race. I wonder how the survey questionnaire translated the word. If it is translated as “jaati” or some such term, i am fairly sure, most Indians would not interpret that word to mean “race”, but they would interpret it as caste or something like that. Also, frankly, most Indians, probably have never met anybody from a different race. So, it is impossible to say if they are likely to discriminate against people from a different race. There is no doubt that, in India there is huge discrimnation based on various factors – caste, religion, gender etc. But the point of the article is that, a meaningless question from a survey is being used to make some very strong conclusion and hence Siddartha’s point is very valid. Max Fisher has over simplified and misinterpreted.

      • I agree. As Mister Mitter pointed out, the central thesis of this exercise is almost as ill conceived as the execution of it. It would be near impossible to formulate a cross-country, let alone global “survey of racism” in this fashion. S4S3, that’s a point well taken. Categories like ‘race’ are so deeply overlaid with other categories like region, religion, caste, gender, class, etc. in India, that it is impossible to meaningfully isolate any one of them and draw conclusions therefrom. For instance, different localities of Delhi would discriminate on the basis of different combinations of class, religion and region; each modulated and tempered to different degrees by the other.

  12. A technical note on how the survey was conducted in India (which even Lakshmi Chaudhry and Sandip Roy seem to have missed in their Firstpost article):

    – the survey questionnaire was translated into ten regional languages before being administered.
    – the Hindi translation of the relevant question asks about whether the respondent would like a person of different “jaati” (“दूसरी जाति के लोग “) as a neighbor. For most Indians, “jaati” translates to caste rather than race (as understood in the Western world).

    So while the survey response doesn’t speak well of the state of casteism in India, it is fundamentally irrelevant to the issue of racism as interpreted by Fisher et al. As you say, race “is impossible to operationalize in a cross-national comparison” and the study of the Indian questionnaire illustrate this perfectly.

  13. It is surprising that the opinions of less than 1800 individuals from a total population of 1.2 Billion Indians could decide that India is the raciest country in the world.

    What about USA, where incidences of intolerance on Black skinned people still occur and against Indian women who wear Bindi on their forehead ? Continuous flow of insults to Indian Gods and Godesses by printing them on many products including sandals/footwears, etc. happens in the most advanced and tolerant society…..

    Funny, Pakistan where every Friday, churches are bombed, Shia Mosques are destroyed, shias are mass murdered, hindu population in 1947 from 24% has come down to less than 1% in 2000, indenture of labour (poor hindus) is still practised by chaining them in the night and supervised with guns in the day time….. is a less tolerant society…????? ha….ha….ha….

    India is still the most tolerant society in the world, considering the weak responses from hindu society despite violent and non-violent aggression going on at all levels especially in the conversion business…. Tolerance in India is a feature since time immemmorial that is why all the persecuted sects in human race during the last known 2000 years have taken refuge in India and made a fear free life, whereas USA / Australia / Canada and Europe have decimated almost all the aboriginals and their civilizations and still look down on the original natives……

    The west and europe can claim some civilized behavior towards other races only during the last century or so. Why the entire Phillipines population had to convert to christianity in the early 1900s when americans were happy and tolerant with their native religious practises ?????

    This kind of survey emanates from the filthy minds who want to tarnish India by whatever means they could put their hands on…….

    • First, your assertion about the “entire Philippines population (converting) to Christianity in the early 1900’s” is nonsense — most of the population had converted to Catholicism under Spanish, not American, colonial rule hundreds of years earlier. The only exceptions were small groups of indigenous persons and Muslims in the Sulu Archipelago and Mindanao. The Catholic Church retains enormous power in the Philippines (to its detriment, I think, but that’s a story for another day) and this cannot be laid at the feet of the US. Second, the argument that India is the “most tolerant society in the world” is also problematic — while it is true that numerous faiths/ethnicities/nationalities do co-exist peacefully for the most part, there are significant problems with Hindu-Muslim conflict, gender-related oppression, attacks on Christians in places like Orissa, and so on. This is not actually a knock on India — any country that diverse is bound to have its communal problems, and there are other countries with far worse issues — but invoking an ahistorical and essentialized “tolerant India” obscures those conflicts that do exist.

  14. It would be interesting to move from this, perhaps necessary, denunciation of shoddy methodology to something like a discussion about prejudice broadly. So brownie points for whataboutery on race, ethnicity, caste, religion and homosexuality as socially-constructed/ inaccurate descriptions of stratification and categorization, with a large slice of nationalistic pride, but I think the survey attempts something useful.

    What would a survey more competently prosecuted show, and wouldn’t that be useful?

    Fischer and someone points out in the comments, usefully, that the results of such a survey may just reflect political correctness. So this, if well done, may be more a map of what countries have embraced the idea that expressing social intolerance is offensive, than a cartography of bullshit.

    I think the knowledge that prejudice is morally wrong, rather than natural, ordained by god, legally and morally acceptable or even promoted, is a very useful first step.

  15. The article you linked to from FirstPost makes my point wonderfully. It suggests that we ought to regret the dilution of ethnically pure neighbourhoods. So much work to do.

  16. “This is an endemic problem across the massive middlebrow “Ideas” industry that has overwhelmed the Internet, taking over from more expensive activities like research and reporting.” –Oh the irony of making this argument in this forum.

    Basically this piece translates to “Oh no, someone produced a dodgy map that makes English speaking, Western countries look comparatively good on a very narrow issue! People should pay no attention to any commentary that doesn’t come from the correct (academic) sources with the correct (progressive, post-colonial) implications.”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s