Center for a New American Security

Stand your ground, Abu M!

The notion that we face a binary choice between 'hard quantitative analysis' and 'faith-based policies' is absurd on its face. The question isn't whether our policies ought to be well-ground in the available evidence, it's about the nature of that evidence and its analysis. And, as you argued before, there's a mounting body of, well, evidence, that quantitative approaches to counterinsurgency are deeply flawed.

The article to which you link is a perfect example of the promises and limits of quantitative approaches. Like their humanistic cousins, they can work reasonably well at assessing and documenting events. So Shapiro and Fair mine public opinion data in order to assess the several factors that correlate with support for militancy. And they're notably modest about the predictive or prescriptive conclusions they draw:
These andings suggest that public support for militant organizations appears to be much more complex than many analysts believe. Respondents in our survey appear to be making rather sophisticated political calculations that are not easily categorized....The implications that follow from this survey do not translate easily into policy action; rather this effort identiaes an empirical agenda to better under- stand and address the roots of support that militancy enjoys in Pakistan.
In short, the example of excellent quantitative work that you offer here looks an awful lot like a humanistic approach dressed up in quantitative clothes. Or, if you prefer, it suggests that quantitative approaches may be valuable when - and only when - they accept the limitations of their methodologies, eschew reductive predictions, and embrace complexity.

But of course, that actually supports your original critique. This sort of paper is the exception that proves the rule. The main problem with quantitative approaches is that they glue on to the messy world of human society a shiny veneer of science. They ape the certainty of physicists in reactions that are vastly less predictable and more complex. This is tremendously appealing to policy makers, who hate uncertainty. But time and again, relying on quantitative social science to provide a foundation for policy has produced policies that are just as limited and reductive as the research on which they are built.

If you're suggesting that this limitation can be overcome by using quantitative analyses as a tool, alongside other humanistic approaches, that's a conclusion I'll heartily endorse. But the notion that quantitative approaches somehow inoculate us against bias or faith-based policy making is chimerical. In fact, all the available evidence points in precisely the opposite direction - excessive reliance on quantitative methodologies tends to blind researchers to the decisions and biases that informed their construction and execution of their studies and so color their findings, and tends to produce excessive degrees of certainty and resistance to constructive critiques. If you want policies grounded in reality, it sure helps to base them on studies which attempt to grapple with reality in all its myriad complexities, and not just a half-dozen easily quantified factors.

AM:

I'll concur with Cynic. (How could one not after his beautifully-written post)? I'd invite you once more to read Stephen Walt's "Rigor or Rigor Mortis," the basis for the book "Rational Choice and Security Studies," in which he takes game theoretic approaches to task for contributing banal conclusions.

I'd also urge you to look at David Laitin's "Comparative Politics: State of the Subdiscipline," in which he argues the disciplines uses three methods: case studies (ie, qualitative methods), formal modeling (ie, game theory), and statistics (self-explanatory). I think disaggregating what and who "quants" are would prove useful to you. (Last night I had dinner with someone for whom running Monte Carlo simulations is fun, fun, fun!)

I think game theory can prove useful as a heuristic device, and can aid one in theory formation. (I base this not off any personal experience as a game theorist, but with my passing exposure to it, and, perhaps embarrassingly, my reading Robert Jervis saying it had helped him compose his work on the spiral and deterrence models.) I also agree with Stephen Walt, though, that it often leads to the trivial. For what little it's worth, in the spirit of incestuous COIN (oops - did I say the wrong word?) blogs, I remember Niel Smith aka CavGuy of SWJ saying he agreed with Walt.

The same problem can exist with statistical work. SNLII's question in re: Lyall and Wilson's paper (and did you see the CAMOS email O'Hanlon sent to Lyall? One, if you're not on the CAMOS list, you should be; Two, it was funny, funny, funny!) was a trenchant one: did anyone think force structure did *not* matter in COIN? To ask a question I think Jack Snyder asked, at least on a syllabus, about Laitin's "Identity in Formation," isn't the good stuff in Lyall and Wilson's article not their attempt to test their hypotheses quantitatively, but their qualitative account of how force structures evolved over time due to various factors? Speaking of Laitin, have you read Nicholas Sambanis's Journal of Conflict Resolution (or Journal of Peace Research?) destruction of his and Fearon's vaunted "Ethnicity, Insurgency, and Civil War" (APSR, Feb 2003 - a journal I believe you said you do not even open)? Basically, Sambanis found that Fearon and Laitin's measures were not robust at all. I think - to finally make sense of this rambling, name-dropping post - that this is where Cynic and I join forces (join hands?): statistics are good for what they're good for, when they're able to be good (ie, to use phrases my friends and colleagues and associates use but which I never really understand, there are no ecological inference problems, etc).

In sum: qualitative work is good and should be falsified when possible, including by numbers, one should separate quants, formal modeling can be useful but can also lead to trite conclusions, the same goes for statistical analysis.

Regards
ADTS

Elf: Amen. Its when they start to make decisions based on metrics without user-input-adjustment that it becomes deadly.

In re: Elf and Fnord:

There was a good article ("Intelligence to Please?") on this some years back by James Wirtz in Political Science Quarterly, about how the body-count mentality contributed to misleading orders of battle in Vietnam.

ADTS

To echo Cynic and ADTS, I would also point out that Lyall's most recent paper makes, in my opinion, fatal error by conflating ethnicity with access to local information, and then overdrawing his conclusions while contradicting (and seemingly ignoring) Kalyvas' data. Such overconfidence and elision of nuance strike me as typical of overly quantitative approaches, and I shudder to think Lyall might have promoted ideas based on that analysis while teaching at CTC-A.

we must to pray about this

Interesting link to these "quant" artistes, but let's respin it a little:

"Fundamentalist Christian militancy in the United States has long stood atop the international security agenda, yet there is almost no systematic evidence about why individual Americans support Christian militant organizations. An analysis of data from a nationally representative survey of urban Americans refutes four influential conventional wisdoms about why Americans support Christian militancy."

"First, there is no clear relationship between poverty and support for militancy. If anything, support for militant organizations is increasing in terms of both subjective economic well-being and community economic performance."

"Second, personal religiosity and support for fundamentalist laws are poor predictors of support for Christian militant organizations."

"Third, support for political goals espoused by legal Christian parties is a weak indicator of support for militant organizations."

"Fourth, those who support core democratic principles or have faith in the United State's democratic process are not less supportive of militancy."

"Taken together, these results suggest that commonly prescribed solutions to Christian militancy—economic development, democratization, and the like—may be irrelevant at best and might even be counterproductive."

The main problem here is that they only talk to urbanites! The situation in rural areas might be quite different. Quantitative analysis based on limited data is just plain dumb - unless there's an answer you've been tasked to produce.

http://www.cnn.com/2010/POLITICS/02/05/obama.cia.memorial/index.html?hpt...

Washington (CNN) -- President Obama is scheduled to attend a Friday memorial service for seven CIA officers and contractors killed in Afghanistan in December.

The service is to be held Friday morning at CIA headquarters in Langley, Virginia.

A suicide bomber killed the CIA officers and contractors, as well as a Jordanian intelligence official, on December 30 at a U.S. base in Khost, in southeastern Afghanistan.

The bomber was within seconds of being searched by security contractors when he detonated his explosives, a former intelligence official with knowledge of the incident told CNN in January.

Two of those killed were contractors with private security firm Xe, formerly known as Blackwater, a former intelligence official told CNN. The CIA considers contractors to be officers.

U.S. and Jordanian officials say the bomber, Humam Khalil Abu-Mulal al-Balawi, had been recruited as a counterterrorism intelligence agent, despite concerns over his extremist views. He was being used in the hunt for a senior al Qaeda figure.

Reva Bhalla, director of analysis for the international intelligence company STRATFOR, said in January that the suicide bombing was "a huge blow, symbolically and tactically." The bombing eliminated so many CIA officers, who can take years to become ingrained in the region.

In addition, the attack showed the ability of the Taliban to penetrate perhaps the most difficult of targets -- a CIA base, she said.

Former CIA official Robert Richer called the bombing the greatest loss of life for the agency since the 1983 bombing of the U.S. Embassy in Beirut, Lebanon, which killed eight agents.

=========================================================================================

The words that set me off are.... "The CIA considers contractors to be officers." The term Case Officer or Officer should only be used for someone who has obtained a college degree.

A majority of Xe Solutions personnel only have a high school education or a GED.

When did Xe / Blackwater Bodyguards starting having IQ Tests, giving them polygraphs, spending several years training its personnel or better yet,....when did Xe's personnel start obtaining PhD's? Over 1/2 of Agency's Case Officers have PhD's and almost all have some kind of Graduate Degree and Language(s) under their belt.

That statement is a slap in the face to every Case Officer out there.... These Xe Solution PS (Protection Specialists) were NOT/NOT very special on that fateful day. They didn't search this Jordanian scumbag until he had already passed through several physical security measures and compromised the well-being and safety of these now deceased Case Officers.

CNN should retract that statement. I spit at you CNN....

But...what if that's what the CIA Public Affairs staff told CNN.

It sounds kind of unlikely that idea occurred to them on their own....

Gee, Xe doesn't have high standards anymore? They used to want mostly special ops, law enforcement, both...although I don't know what their formal educational requirements were or are. However if you happen to have experience in the field and have completed the professional education and hurdled over the military/SOF bars - pretty high - how relevant is a college degree? Which BTW doesn't mean all that much anymore, it basically means what a High School Degree used to mean.

And has it been established at what point Xe gained control or responsibility for searching the informant? Who got him waved into the base, CIA or Xe?

People love to hate PMC's, but here if was the "professionals" who screwed up.

I do believe the GED's over at Xe, had they been tasked with vetting could have been capable of finding Mo Boom on the net and might have been a tad bit more careful.

I also believe that if we took your average NJ high school class and used them to replace the CIA, we'd probably get better intelligence. Then again, they could also replace most of our DC Supreme Soviet leadership with High Schooler's and get better results.

If it's PMC hate - wrong road. They are actually a big part of the answer now - if ever there was a need for us to dust off letters of Marque and Reprisal this is it, and they're the ones who should be handed the letter.

1) I don't know enough about terrorism to conjecture whether the main backers of it are people of substance (which is the same as people of wealth?) in society. It certainly seems plausible, though. What comes to mind is, of course, the bin Laden family's wealth. That said, terrorism is a broad concept, who are terrorists is a matter of contention, and (again) I'm merely speculating.

2) I don't know if real democracy is a threat to terrorism. Again, terrorism is a broad concept, ill-defined and subjective. I agree that entities (e.g., states) with good civilian-intelligence relations (Michael Handel's term, not mine) would seem less inclined to use intelligence services (and/or perhaps would use them for intelligence gathering strictly construed, rather than paramilitary clandestine operations). Not that this relates to your point directly, but there's been some work done (Erica Chenoweth?) on whether terrorism is dealt with better by democracies (e.g., perhaps hamstrung by the rule of law).

3) Democracy can definitely be a threat to foreign business interests. Recall CNOOC's attempt a few years ago to make an acquisition of an American firm. I suppose an argument along these lines would run something like this: democracy creates multiple veto points, each of which hinders the potential of foreign business interests to operate within a country. Also, should you wish to go this route, you might wish to read "U.S. Politics and Greater Regulation of Inward Foreign Direct Investment" by C.S. Kang in "International Organization" (a good non-quantitative article!). At the same time, autocracies may be more arbitrary and capricious toward foreign business interests precisely because they lack veto points, and are thus less "sticky." If foreign business was not a tricky business, political risk analysts (Eurasia Group, Oxford Analytica, Economist Intelligence Unit, etc.) would make less money.

ADTS

Niel:

I don't have a critique. I was just (1) referencing an email sent over a listserv (CAMOS) complimenting Lyall and criticizing Princeton, sent by Michael O'Hanlon (!), and (2) referencing that I thought you had composed such a critique(!).

I'm not a quant, and I find Lyall and Wilson good at the skill of "confessing [their] sins" so that reviewers don't disqualify them accordingly. And I hadn't read the article beforehand. But I just read it briskly on my laptop, sans the case study portion (which I nonetheless address), and here are some barebones thoughts:

The paragraph on the top of page 71 contains a huge assumption (insurgency is ahistorical).

Outcome on page 71 also contains a huge assumption – it’s coded from the incumbent’s point of view.

Their coding procedure also seems subject to question – see second full paragraph on page 72. How does one measure political processes?

Page 73: was the foraging interaction population-centric or “population-coercive?” Is it the structure of the force or how it behaves? Also, is it foraging per se, or is the fact that there simply were no armored vehicles? (The “foraging plus railways” result is an interesting one.) Furthermore, doesn’t Van Creveld argue that a lot of movement – and hence presumably not a lot of information-gathering – occurred because armies had to move a lot?

Biddle writes what he writes (top of page 75), but does he write it in the context of colonial wars, or in the context of a “modern system” of conventional war? (Middle of page 75)

The modeling of patrons’ militaries was symbolic – eg, Nigeria getting F-16s (Priest, “The Mission”) – rather than reflecting actual practice; states adopted formal institutions rather than adopting informal practices.

Footnote 28: I’m not sure Kalyvas would agree, and he’s excluded. Additionally, I’m struck by the absence of Merom on footnote 46 from this account: he’d argue brutality is an option (and I think Ivan Arreguin-Toft is working on brutality – “barbarism” – in irregular warfare) but that it’s incompatible with certain regime types.

Also, this paper doesn’t account for selection effects – the insurgencies that didn’t occur.

Pages 79-80: I like what I’m reading, but couldn’t Fordist/post-Fordist efforts be applied to COIN, too?

Pages 79-80: Incidentally, weird (although far less so than before) for a quantitative paper like this one to rely on arguments regarding international norms.

Page 81: see first full paragraph for my critique of the case study portion.

Bottom of page 81: they’re not using audience costs in the traditional sense (Fearon mid-90s), even though they admittedly are using it in a way that makes sense. (Interesting result page 91)

Pages 82-83: they concede three alternative explanations might all be correct! (Confession of sin)

Footnote 58: they use 1917 as their cutpoint. THIS IS HUGE!!! Can they justify it? Why not 1865, say? (Confession of sin)

Pages 84-85 What if percent mechanized is not just relative but absolute – that is in part the critique of the CAPS program – there were not enough QRF assets to safely garrison villages? OK – helicopter as an indicator – but were helicopters maintained, used (SVN Army?)? Again, are postcolonial armies the same as colonial armies?

*All their indicators seem somewhat fair (overall), but you can always pull apart indicators.*

Trade as an indicator of sensitivity to international pressure – I thought we were talking about norms, not trade! How about number of international treaties signed, or NGOs hosted, or International NGOs hosted?

For Footnote 67, note Fearon and Laitin don’t measure swamps, only mountains (what can be measured) (and see result on page 88), and also, check out Brian Michael Jenkins on urban insurgency (there’s also a SAAMs paper on rural versus urban insurgency I can give you a URL for).

For Distance, as a sidenote I think John Geis (USAF) has written on this, and found it doesn’t hold – I don’t recall what results he did or did not receive, but it wasn’t the predicted one (ie, the one found here).

Is there a model of “Mech” AND “Heli”?

I skimmed lightly over the case study, and looked at your questions, and would boil it down to “omitted variable bias,” and think one could construct an article around it as an alternative explanation: Armies do well or poorly based on their LEADERSHIP, which is a function of (professional military education, institutional experience and culture (see Elizabeth Kier in addition to John Nagl,), etc).

Page 102 – fallacy of composition – are these all civil wars? Page 102 – insufficient example to periodization. (Confession of sin)

ADTS