The Shirt of Nessus: The Rise and Fall of Western Counterinsurgency

We still assume that if correctly managed, counterinsurgency can succeed anywhere. It is a dangerous assumption, for it may lead policymakers to commit American lives, lucre, and prestige to causes better analysis might have revealed to be chimerical.
— D. Michael Shafer, Deadly Paradigms: The Failure of US Counterinsurgency Policy

In a speech at the Australian Defence College in Canberra in February 2007, the Chairman of the United States Joint Chiefs of Staff, General Peter Pace, posed the question: “How do the West’s fast-food nations fight a one-hundred-year war against Al-Qaeda and its Islamist allies?” Even as Pace spoke, the response was already well under way in the halls of power in Washington and London: employ armies in long-term counterinsurgency campaigns over years, if not decades, in order to bring about stability in Iraq and Afghanistan. Yet, there is probably no more unpopular form of war for a liberal democracy than counterinsurgency. The West’s record in fighting modern insurgents from the Cold War era to the age of globalisation is characterised by multiple political reverses.

This article seeks to explain the swift rise and fall of Western counterinsurgency between 2004 and 2014. This is an important task to undertake because, even though the West has now abandoned its brand of post-9/11 counterinsurgency, our Islamist opponents from Al-Qaeda in Afghanistan through Al-Shabaab in Somalia to Islamic State in Iraq and Syria continue to wage modes of insurgent warfare. We now face the difficult task of crafting an alternative way of war to fight guerrillas and militia forces and, unpalatable though it may be, we must seek to learn lessons from our recent experience with counterinsurgency.

Three areas are covered. First, to establish a historical context for analysis, the evolution and eclipse of the modern doctrine of counterinsurgency during the Cold War is briefly assessed. Second, the unexpected and enthusiastic revival of counterinsurgency which began after 2004 in Iraq is critically examined. Finally, some of the main reasons for counterinsurgency’s extraordinary fall from grace between 2011 and 2014 are explored and some propositions are advanced on how the West might seek to confront irregular opponents in the future.

Classicists and modernisers: The rise and fall of Cold War counterinsurgency

In order to understand the problems encountered in Western counterinsurgency over the past ten years it is necessary to snapshot the evolution of this form of warfare. Although its historical roots can be traced to nineteenth-century doctrines of colonial warfare, modern counterinsurgency is really a phenomenon of the Cold War between 1949 and 1979. This period witnessed a unique intersection between the end of Western colonialism; the rise of communist people’s war stemming from Mao Zedong’s victory in China; and the collision in the emerging Third World between rival systems of capitalist and communist modernisation.

The confluence of these forces encouraged the growth of Cold War revolutionary insurgency movements which combined nationalist zeal with modern communist organisation and age-old guerrilla methods. From Asia through Latin America to Africa, novel forms of modern insurgency—forms that took inspiration from the theories of Mao Zedong’s protracted people’s war and later, from Che Guevara’s Cuban foco theory (guerrillas as detonators of revolution) began to proliferate. Rural guerrilla operations—once dismissed by orthodox Marxist-Leninist theorists as repositories of peasant backwardness immune to the class struggle—were transformed by Chinese and Cuban dialectics into new doctrines of revolutionary warfare designed to win power for the masses of the countryside.

Western strategic studies specialists are often inclined to see the first half of the Cold War as the era of deterrence and nuclear stalemate, a “long peace” underpinned by a balance of terror. Yet for those concerned with armed struggles beneath the nuclear and conventional warfare thresholds, these years were part of a “long war” waged in the shadows against revolutionary insurgent movements.

Two pioneering schools of thought on counterinsurgency emerged between the late 1950s and the early 1970s: the Anglo-French classical school and the American modernisation school. The Anglo-French classicists were mainly theorist-practitioners with strong military backgrounds. Their ideas were shaped by the challenge of confronting the nexus of colonial retreat and communist guerrilla advance as the age of empire disappeared. Common to the classical school was the notion that the essence of counterinsurgency involves an armed struggle between an incumbent government and an insurgent movement for the political allegiance of the population. Such a contest is sometimes described as winning the “hearts and minds” of the people—a phrase often attributed to the British general, Sir Gerald Templer, during the Malayan Emergency of the 1950s. It is a misleading notion because the primary problem to be resolved in any counterinsurgency is less the winning of “hearts and minds” than establishing effective security on the ground in a manner that facilitates political progress. A balance needs to be struck between using coercion and seeking co-operation; between “search-and-destroy” missions against guerrilla cadres and mounting “secure-and-hold” operations to protect the target population against insurgent influence.

The Anglo-French theorists who wrestled with counterinsurgency doctrine included such figures as Robert Thompson, Frank Kitson, Roger Trinquier and David Galula. Collectively, they authored the corpus of knowledge now often described as classical counterinsurgency. Thompson, a former Royal Air Force officer and Malayan colonial administrator, published his seminal study Defeating Communist Insurgency in 1966. The book was a meditation on how Britain had defeated the Malayan communist insurgency between 1948 and 1960. Thompson advanced his famous “five principles” for achieving success. First, a government must have a clear political aim; second, it must function according to the law; third, an administration must possess an overall plan to guide unity of action; fourth, and arguably most important of all, a government must give priority to defeating the causes of political subversion, rather than simply physically eliminating guerrillas. Finally, counterinsurgency forces must secure their base areas and possess sufficient control to be effective.

In Malaya, the application of these principles by a retreating British colonial administration led to the defeat of communist subversion and “winning politically” through the creation of a non-communist independent state. Based on this success, Thompson’s principles appeared to offer a universal set of guidelines for aspiring counterinsurgents to direct their military efforts towards achieving favourable political solutions. By the 1970s, Defeating Communist Insurgency became the standard counterinsurgency text for British-style militaries from Australia to Rhodesia and was influential in shaping doctrine from India to America.

Thompson’s pioneering work was extended by Frank Kitson, a British Army officer with long experience in fighting guerrillas in Kenya, Malaya and Northern Ireland. Kitson’s books—Gangs and Counter-Gangs (1960), Low Intensity Operations (1971) and his autobiography, Bunch of Five (1977)—dwelt on the roles of politics, intelligence, civil-military co-ordination and psychological operations in defeating insurgency. In Bunch of Five, Kitson reinforced Thompson’s message that achieving political success was the key determinant in counterinsurgency. As he sagely put it, “there can be no such thing as a purely military victory because insurgency is not primarily a military problem”.

The key French contributions to classical counterinsurgency became Roger Trinquier’s Modern Warfare: A French View of Counterinsurgency (1961) and David Galula’s Counterinsurgency: Theory and Practice (1964). Both books were deeply influenced by France’s colonial warfare heritage and its post-Second World War experience in Indo-China and Algeria. Although it was not recognised at the time, Trinquier was the originator of the idea that struggles amidst populations represented the post-nuclear future of armed conflict. When the British general Sir Rupert Smith posited in his 2005 book The Utility of Force: The Art of War in the Modern World that interstate industrial war had been replaced by “war amongst the people” his notion was largely a restatement of Trinquier’s concept that “modern war” involves fighting amidst populations. In 1961 Trinquier argued that modern war “is now an interlocking system of actions—political, economic, psychological, military”. Insurgents would seek to wrest power not by pitched battles but by la guerre revolutionnaire—the use of subversion, propaganda, violence and terror. Trinquier believed that the primacy of politics in insurgency could be exploited by lightly-armed combatants to checkmate the conventional military supremacy of advanced countries. In a central insight, he lamented:

The traditional army, having at its disposal large numbers of trained troops and an abundance of modern materiel in the final analysis is completely incapable of overcoming a practically destitute enemy whose leaders and men have received only rudimentary military training. Incredible as this seems, it is nonetheless a bitter reality … Our [French] army has not succeeded in adapting itself to a form of warfare the military schools do not yet teach.

David Galula’s contribution to counterinsurgency has arguably proven to be the most lasting of the classical theorists. This has occurred largely because many in the American military in Iraq embraced Galula’s work and attributed success in the 2007 counterinsurgency “surge” to his teachings. Galula’s writing contains important lessons for any student of counter-guerrilla warfare. For example, he highlights that effective counterinsurgency is, to use today’s terms, often “population-centric” (focused on securing and controlling a given population) rather than “enemy-centric” (focused on defeating guerrillas militarily). This premise leads to Galula’s second proposition: namely that the real task of the counterinsurgent is an armed competition to secure the population against insurgent subversion. Galula teaches that a war against insurgents is 80 per cent political and only 20 per cent military, requiring a reformist impulse and unity of civil-military effort from any incumbent government. All military action must follow this “80:20 rule” and seek to “afford the political power enough freedom to work safely with the population”. Of all the Anglo-French classical theorists, Galula continues to attract the most serious analysis today.

The second major Western counterinsurgency school that developed during the Cold War era was that of the American modernisers, whose efforts ran parallel to those of the European classicists. In contrast to the Europeans, the impulses informing US counterinsurgency by the early 1960s were firmly post-colonial, developmental in character and often driven by civilian theorists. Many of these civilians were social-science intellectuals with backgrounds rooted in the traditions of Max Weber and Emile Durkheim. They included the political scientist Lucian Pye and the policy specialists Roger Hilsman and Walt Rostow.

The modernisers possessed powerful political patronage from President John F. Kennedy. He had a strong personal interest in counterinsurgency as a method of underpinning efforts to portray American capitalist development as a superior model to that offered by communism. In January 1962, Kennedy created the Special Group CI (Counter-Insurgency) “to insure [sic] proper recognition throughout the US Government that subversive insurgency (‘wars of national liberation’) is a major form of politico-military conflict equal in importance to conventional warfare”. In this way, counterinsurgency became an integral part of America’s Cold War arsenal to win “the revolution in modernisation” occurring in the new countries of the Third World.

Between 1961 and 1963, America’s focus on counterinsurgency helped to transform South Vietnam from an Asian strategic backwater into an anti-communist front line. In effect, modernisation theory, anti-communist ideology and counterinsurgency coalesced in Vietnam, which became a vast “New Frontier” laboratory for counter-guerrilla specialists, intelligence operatives, defence pundits, military contractors, research consultants and aid experts all seeking to build Saigon’s resilience. Observing the swift rise of American counterinsurgency in early 1963, the leading French scholar on Vietnam, Bernard Fall, remarked wryly, “guerrilla experts [in Washington] are like streetcars; there is one coming by every ten minutes”. To the “best and the brightest” in both the Kennedy and Johnson White House, American counterinsurgency was part of a transformative social engineering crusade to stimulate nation-building in the underdeveloped world. Communist insurgents were seen as flies in the ointment of progress. Rostow called them “the scavengers of the modernisation process” to be isolated or eliminated as the South Vietnamese people transitioned from an agrarian society to a nation-state.

Yet despite the impressive Western theorising from both the Anglo-American classicists and the American modernisers, the practice of Cold War counterinsurgency proved bitterly disappointing. French counterinsurgency led to military success in Algeria in the late 1950s but at such a cost in bloodshed and torture that it created electoral revulsion and the conditions for political defeat in 1962—a stunning illustration of Trinquier’s “bitter reality” of a powerful army losing to an inferior military enemy. Meanwhile, the five principles of Thompson’s “Malaya model” as a universal formula for political success in counterinsurgency proved difficult to implement elsewhere. Colonial-style regimes keen to stay in power, such as those in Rhodesia and Portuguese Africa, were interested only in military, not political, solutions to insurgency and paid lip service to Thompson’s principles. In 1979, fresh from observing the Rhodesian government’s overwhelmingly military approach to counterinsurgency, Thompson came to the following conclusion:

The basic winning formula for an insurgency is as follows: if an insurgent movement can, at a cost which is indefinitely acceptable, impose costs on a government which are not indefinitely acceptable, then, while losing every battle, it is winning the war. In short the [counterinsurgency] army wins all the battles and is still defeated in the end.

It is this paradoxical formula of “winning all the battles, but losing the war” that eventually made Cold War Western armies so distrustful of counterinsurgency doctrine. While one can deduce other reasons for the demise of Western counterinsurgency in the 1970s—notably the end of decolonisation and the revival of conventional war through precision weapons—it was in Vietnam that the paradox of “winning battles but losing the war” was most vividly illustrated. The failure of America’s “counterinsurgency as modernisation” doctrine in Vietnam had the general effect of discrediting the main tenets of Western counterinsurgency for the next thirty years. As Michael Latham reflects in his 2000 book, Modernization as Ideology, “unfamiliar with Vietnam’s culture, most American policymakers never understood the [Viet Cong] revolution against which they fought”. A hopelessly weak and corrupt South Vietnamese regime failed to resist the Viet Cong and its backers in North Vietnam. As a result, all American military efforts became futile, and intervention became a political debacle for Washington. As Charles Maechling, Jr., Lyndon Johnson’s former Director of Counterinsurgency, lamented in 1969, a corrupt Saigon government turned the American counterinsurgency campaign in Vietnam into “the old horror of responsibility without authority elevated to the plain of high strategy”.

Not surprisingly, by the 1980s from its highpoint in the 1960s, counterinsurgency went into sharp decline in the American and Western European militaries. In particular, the US military rediscovered conventional warfare and became enamoured of the high-technology doctrine of AirLand Battle. The very term “counterinsurgency” disappeared from American military circles and was replaced by the notion of “low-intensity conflict” (LIC) and later by other concepts such as “operations other than war” (OOTW), “asymmetric warfare” and “full-spectrum operations”. In July 1990 two leading scholars, Raj Desai and Harry Eckstein, writing in the journal World Politics, delivered a verdict on Western counterinsurgency that seemed, as the Cold War slipped into history, to be an obituary for an entire form of armed conflict:

Nothing in contemporary military writing is more pathetic than the voluminous attempts to find viable counterinsurgency doctrines. And nothing in recent violent conflict has been more cruel or inefficient than the practice of the doctrines. Our [Western] failure to grasp the very nature of the phenomenon is most evident in the frequent reduction of insurgency to mere warfare—of a kind: guerrilla, irregular, remote-area, low-intensity [sic] and the like.

Western amnesia on counterinsurgency was only encouraged by the 1991 victory of the US-led coalition over Iraq in Operation Desert Storm. For the American military, the Gulf War was everything Vietnam was not: swift in execution and decisive in outcome. Uniformed professionals became enthused by a “shock and awe” mentality expressed through the construct of a Military Technical Revolution (MTR) or Revolution in Military Affairs (RMA) in which information-age digital networks and precision engagement would confer indefinite Western military supremacy.

Yet, in many ways, the First Gulf War proved misleading on the future shape of military operations. The pattern of warfare in the 1990s was dominated less by conventional conflict than by irregular wars from Somalia through the Balkans to Chechnya, often involving transnational violence by non-state groups. In October 1996, the growing disparity between the theory and the practice of warfare caused the Commandant of the US Marine Corps, General Charles C. Krulak, to observe famously:

the asymmetric threats of the day after tomorrow will call for more than forces that can simply fight Desert Storm better. The future [of war] may well not be “Son of Desert Storm” but rather “Stepchild of Somalia and Chechnya”

Revival: The New Counterinsurgency Era, 2004-14

The West’s “son of Desert Storm” vision was shattered by the Al-Qaeda attacks on the United States on September 11, 2001. Between 2001-02 and 2003 respectively, American-led coalitions quickly overthrew the Taliban theocracy in Afghanistan and then the Ba’ath dictatorship in Iraq. Yet American-led forces found that regime change in Afghanistan and Iraq were the easy parts of both campaigns. Through a mixture of conventional military hubris, ethnocentrism and political miscalculation, stability in both countries disappeared into a whirlpool of ethnic chaos and xenophobia that bred an unexpected and almost forgotten form of warfare: insurgency.

Western forces found themselves trapped in two large-scale guerrilla campaigns for which there was no readiness or institutional knowledge. The first was against a resurgent Pashtun Taliban movement operating from neighbouring Pakistan; and the second was waged by Sunni Iraqis fearful of a US-backed Shia-dominated government in Baghdad. By 2004, faced with a strategic quagmire in Iraq and fierce resistance from well-armed Sunni insurgents, American-led coalition forces scrambled to rediscover long-lost principles of irregular warfare. This process sparked a military revolution that was truly astonishing in its speed and reach and which within a few years resulted in a “new counterinsurgency era”. In the words of Thomas Rid and Thomas Keaney in their 2010 book, Understanding Counterinsurgency, “a conceptual reorientation of gigantic proportions took place inside the US armed forces … The debate’s range of ideas and the number of its publications assumed almost encyclopedic proportions.”

By 2008, a counterinsurgency industry to rival that of the 1960s stalked the corridors of power in Washington. Three identifiable schools of thought developed: a neo-classical school; a stabilisation school; and a third that emphasised global features. The neo-classical school exemplified by American soldier-scholars such as John Nagl and Robert Cassidy set about the business of recovering the almost forgotten work of the classical Anglo-French counterinsurgency thinkers. The aim was to improve America’s institutional capacity to defeat guerrillas by returning to the teachings of the Cold War masters. The stabilisation school, often composed of State Department theorists of neo-liberalism and globalisation, focused on reconstruction, economic aid and state-building as a cure for the new brand of Islamist insurgency. Many stabilisation adherents took their cue from Francis Fukuyama’s 2004 work, State-Building: Governance and World Order in the 21st Century and its Hegelian “end of history” re-reading of 1960s modernisation theory. Fukuyama’s central proposition was that “September 11 proved that state weakness constituted a huge strategic challenge” to the West and must be addressed by intervention to facilitate “host nation” capacity-building.

The stabilisation school recycled many of the modernisation ideas of the Vietnam years to argue that counterinsurgency could again be part of politico-economic development of fragile states—this time through “Military Support for Stability, Security, Transition and Reconstruction (SSTR) Operations”. In December 2005, President Bush released National Security Directive 44, “Management of Interagency Efforts Concerning Reconstruction and Stabilization”, designed to build an interagency structure. By 2009, the British military had produced a variant on what it called the “comprehensive approach” to intervention operations in the form of a manual titled Security and Stabilisation: The Military Contribution, which viewed overseas counterinsurgency operations by the United Kingdom as the military component of “wider stabilisation” and aimed at improving host government legitimacy by “military assistance to security and development”.

The third school of global counterinsurgency was at once the most innovative and ambitious. Its proponents included the Australian soldier-scholar David Kilcullen and John Mackinlay, a former British Gurkha officer. Both figures highlighted the novelty of insurgent information networks in order to try to explain the increasing interaction between transnational, local, religious and criminal-narcotic actors in warfare. Their central proposition was that the West faced a long-term, globalised form of Islamist insurgency empowered by global-local connectivity in which transnational insurgents infiltrate and exploit states with weak governance such as Afghanistan, Iraq, Yemen and Somalia. Global and local interaction, they postulated, produced matrices of diverse irregular actors such as that between Al-Qaeda and the Sunni Iraqis and between Al-Qaeda and the Quetta Shura Taliban in Pakistan.

In his 2009 book, The Insurgent Archipelago: From Mao to bin Laden, Mackinlay noted how Maoist insurgent techniques had been adapted to a new, hydra-headed form of interactive Islamist insurgency that was, and remains, far removed from the monolithic Maoist-style national liberation movements of the Cold War era. Transnational networks permitted access to recruits, finance and technology from Muslim communities in an “insurgent archipelago” running globally from Kandahar to Cambridge. Similarly, in his 2009 work, The Accidental Guerrilla: Fighting Small Wars in the Midst of a Big One, Kilcullen suggested that the struggle actually represented a Salafi transnational insurgency with both global and local features. Such a struggle was networked and required a careful analysis of conflict ethnography and local culture in affected countries. Kilcullen warned that, without such knowledge, heavy-handed Western intervention risked creating many “accidental guerrillas”—local actors who might become disaffected by a foreign presence and so be drawn to the Salafi cause.

Given the intellectual ferment that stemmed from the intersection of these three new schools of thought, a legion of Western counterinsurgency champions, nicknamed the “COINDinistas” by critics—a play on the abbreviation for counterinsurgency (COIN) and the leftist guerrillas of Nicaragua (Sandinistas)—appeared in the American military and think-tank community. Like the spartoi of Greek mythology they appeared to have suddenly sprung from dragon’s teeth and they were quick to cultivate a Jesuitical warrior-scholar mystique both in the field and in the halls of Washington. The COINdinistas proved to be great advocates of British pioneer theorist Charles Callwell’s dictum in his 1906 book, Small Wars, that fighting irregulars represented “an art in itself”. They became fond of quoting T.E. Lawrence that “guerrilla warfare is more intellectual than a bayonet charge” and suggested pace Callwell that counterinsurgency was—unlike conventional operations—“the graduate level of war”. Their hubris was allied to a relentless belief in the mastery of special irregular warfare skills—cultural, linguistic, anthropological and sociological—needed to successfully operate in Arab and Afghan populations. In his 2011 memoir, Cables from Kabul, Sherard Cowper-Coles, the former British ambassador to Afghanistan, vividly portrays the “new apostles” of counterinsurgency:

Like its first cousin, stabilisation, counter-insurgency, or COIN, has acquired some of the characteristics of a cult. Disciples speak of its properties with evangelical fervour. There are different routes for gaining admission to its mysteries, but most involve field experience in a combat zone, recently Iraq or Afghanistan, and a period of postgraduate study of uneven rigour. Qualifying involves at least as much faith as works.

The rise of the COINdinistas was symbolised by the rapid promotion of General David Petraeus, a charismatic graduate of Princeton who had written his PhD on the French experience in Indo-China. Petraeus, who viewed the French theorist David Galula as “the Clausewitz of counterinsurgency”, led an intellectual charge to develop new American doctrine. Under his stewardship, by December 2006 a new and unusually well-written joint US Army and Marine Corps Field Manual 3-24, Counterinsurgency, emerged. FM 3-24 was heavily influenced by the COINdinista neo-classical school and their reading of Thompson and Galula. Few military doctrine manuals in modern history have received the popular and critical attention lavished on this publication. Within two months of its release, the document was downloaded from the internet two million times and the manual was published commercially by Chicago University Press. Moreover, the status of FM 3-24 was only enhanced when Petraeus went on to command the famous “surge” in 2007 that turned the tide against the Iraqi Sunni insurgency in America’s favour.

In the space of three years between 2004 and 2007, as necessity spurred rediscovery, counterinsurgency, and its new intelligentsia led by such figures as Nagl and Kilcullen, was suddenly and improbably back in the mainstream of official American military thought. A favourable momentum continued and, in October 2008 in Field Manual 3-07, Stability Operations, non-combat state-building activities were declared by the Pentagon to be equal in importance to major combat operations—a testament to the rising influence of the neo-liberal stabilisation advocates. The years 2009 and 2010 proved to be the high-water mark of the new counterinsurgency era. In January 2009, a special inter-agency US Government Counterinsurgency Guide defined counterinsurgency as those “comprehensive civilian and military efforts taken to simultaneously defeat and contain insurgency and address its root causes”. The Guide blended elements from the neo-classical, stabilisation and global schools of thought and sought to integrate anti-guerrilla techniques with the security, governance and development measures required for state-building.

Many of these ideas seemed to be given flesh in December 2009, when—hoping to transfer American success in Iraq to a rapidly-deteriorating Afghan security situation—President Barack Obama ordered a 30,000-strong American troop “surge” into Afghanistan. Moreover, when the Pentagon’s Quadrennial Defense Review was published in February 2010 it reflected the elevated status counterinsurgency had assumed in American military doctrine:

Stability operations … counterinsurgency and counter-terrorism operations are not niche challenges [or] a transitory or anomalous phenomenon in the security landscape. On the contrary, we must expect that for the indefinite future, violent extremist groups, with or without state sponsorship, will continue to foment instability and challenge US and allied interests.

Yet despite widespread advocacy and apparent official acceptance, the entire American-led counterinsurgency house of cards collapsed just as spectacularly as it had arisen. Unlike Iraq after 2007, Afghanistan came to defy any swift Western counterinsurgency “surge” solution, so presenting the Obama White House with the prospect of an endless and expensive conflict. Official disillusionment with the COINdinistas was quick to emerge. In September 2011, the US Army began to abandon “persistent conflict” (code for large-scale counterinsurgency) and stability operations—in favour of a new doctrine called unified land operations. The latter doctrine returned the US Army to a traditional focus on combined arms warfare as the foundation stone of the American profession of arms.

By the end of 2011, the US proclaimed strategic success in Iraq and withdrew militarily from the country while announcing it would do the same in Afghanistan at the end of 2014. The leading American counterinsurgency generals, Stanley McChrystal and David Petraeus both fell rapidly from grace—being dismissed from service and transferred to non-military positions respectively. Unlike the 2006 version of FM 3-24, an updated 2014 manual published under the title Insurgencies and Countering Insurgencies engendered little interest or discussion outside specialist circles. Finally, in March 2014, and in sharp contrast to its predecessor document, the Pentagon’s Quadrennial Defense Review effectively removed counterinsurgency and stabilisation from future American force development considerations. The publication states:

Although our forces will no longer be sized to conduct large-scale prolonged stability operations, we will preserve the expertise gained during the past ten years of counterinsurgency and stability operations in Iraq and Afghanistan. We will also protect the ability to regenerate capabilities that might be needed to meet future demands.

In short, even though irregular challenges continue to fester and increase, the evangelical COINdinista brand of counterinsurgency practised since 2004 is no longer seen as effective. The swift rise and fall of Western counterinsurgency over the past decade deserves the closest analysis. We need to remember that Islamist insurgents show no signs of flagging and are quick to equate Western military withdrawal on any battlefront as a propaganda victory towards their declared aim of hoisting “a black flag over the White House”. The remainder of this article is devoted to examining some of the main reasons for the demise of the “new counterinsurgency era”.

Reasons for 21^st Century Western Counterinsurgency’s Failure and Implications for the Future

Three principal reasons can be identified for the demise of Western counterinsurgency’s “second coming” in the twenty-first century. First, despite an avalanche of academic and professional publications on the subject since 2004, the ability to relate counterinsurgency in terms of its military and civilian techniques to a broader strategic notion of irregular warfare proved elusive. Second, and related to the above, the elegance of theory again failed to match the reality of practice. As a result, many of the mistakes from the 1960s and 1970s have been repeated between 2004 and 2014. Finally, over the past decade, many counterinsurgency experts (and not a few politicians) have erroneously promoted counterinsurgency—a set of operational methodologies—as a strategy in its own right.

The first reason for the demise of Western counterinsurgency concerns the multiple dynamics involved in this field of armed conflict. Effective counterinsurgency requires not just soldiers but a host of civilians ranging from political leaders through police and civil administrators to assorted aid workers in a comprehensive, “whole of government” and inter-agency effort. Since 2004, there has been little progress in mastering this unified civil-military requirement, simply because there is no common Western approach available. While soldiers tend to concentrate on the military requirements of counterinsurgency for which there is doctrine, for many civil agencies the challenges in Afghanistan and Iraq are often about non-military stability operations and humanitarian aid amongst indigenous populations. Yet in this sphere there is no agreed doctrine to integrate civil-military efforts, guide policy or to evaluate progress. In 2007, the American analyst James Wirtz, writing in a study entitled The Interagency and Counterinsurgency Warfare: Stability, Security, Transition and Reconstruction Roles, observed, “the first stumbling block encountered in the effort to undertake a net assessment of ongoing stability operations is that scholars, soldiers, and policy makers alike, lack a shared understanding of how stability operations actually produce stability”.

There is often considerable conceptual ambiguity on how counterinsurgency and stability operations actually relate to each other. In the decade between 2004 and 2014, the USA sought to fuse combat and non-combat operations together by articulating a sequence of actions known as “shape-hold-clear-build-transition” using both troops and civilian experts to secure disaffected areas of Iraq and Afghanistan. It quickly became evident that, in such a complex activity, measures of effectiveness to evaluate operational progress meant different things to soldiers and civilians. Such a situation is a testament to the conceptual difficulties that plague the field of irregular warfare. As early as 2001, in their major study Warfare and the Third World, Robert E. Harkavy and Stephanie G. Neuman suggested that, unlike nuclear weapons or conventional conflict, irregular conflict remains the most under-theorised area in Western strategic studies. As they put it, in the study of irregular conflict, “there is no emerging futuristic paradigm equivalent to what is wrought by the advent of [the] MTR/RMA; and nuclear weapons; [and] for conventional warfare”. Despite the volume of publication on counterinsurgency and stabilisation, this situation has not changed in 2014 and a comprehensive understanding of the civil–military relationship in irregular conflict continues to remains elusive.

The second problem in the eclipse of Western counterinsurgency concerns an overemphasis on large-scale, direct intervention based on Galula-style “population-centric” methods. In Iraq and Afghanistan, as in Vietnam, such direct intervention simply led to escalating military commitments whose success was hostage to the dilatory policies of weak host regimes incapable of winning broad popular support. As a result, a “population-centric” approach to counterinsurgency has proven merely to be a prescription for placing the burden for political progress firmly on a foreign coalition rather than an indigenous government.

In Afghanistan, in particular, Coalition forces assumed that their “nation-building” objectives were aligned with benevolent policies emanating from the elites within the Hamid Karzai government. This was an erroneous assumption since the government in Kabul was, and remains, a kleptocracy pursuing narrow ethnic policies aimed at staying in power rather than ending a war by political reconciliation. Western countries thus succumbed to a wishful ethnocentrism based on the illusion of congruent values. In a real sense, the fate of Western counterinsurgency in Afghanistan resembles that of Hercules in Greek mythology. When the all-powerful Hercules naively put on the shirt of the centaur, Nessus, he discovered that instead of giving him comfort it was a poisoned garment that infected his flesh and led him to hurl himself in agony onto a funeral pyre. By putting on Kabul’s poisoned shirt, the International Security Assistance Force (ISAF) became tainted by association with a narrowly-based and predatory regime. Given the malfeasance of Karzai government officials and the criminal activities of regional warlords, Coalition forces found it difficult to win over the population against assorted insurgents. As a result, Western assistance did not increase the Afghan government’s capacity and legitimacy but simply helped to extend the scale of its corrupt practices.

The most extraordinary feature in all of this is that none of these problems was new to the American experience of expeditionary counterinsurgency. Indeed, despite being separated by over forty years, there is a striking similarity between the reminiscences of two leading American policy-makers on the situations in Vietnam and Afghanistan. In 1987, Charles Maechling, Jr., the Johnson administration official in charge of counterinsurgency in Vietnam wrote of the US effort in South-East Asia during the 1960s:

[American counterinsurgency] in theory failed in practice since it had to be implemented by an unpopular, unrepresentative local regime … The presumption by the Kennedy and Johnson administrations in supposing that middle-grade US Army officers and civil servants from the American heartland could create a viable rural society … in the middle of a civil war is staggering. There was no way for the Americans to get beneath the surface of Vietnamese life.

In an eerie parallel, in September 2013, in an article in Foreign Affairs, General Karl W. Eikenberry, a former ISAF commander and US ambassador in Kabul, reflected on the American experience in South-West Asia:

It was sheer hubris to think that American military personnel without the appropriate language skills and only a superficial understanding of Afghan culture could, on six- or 12-month tours, somehow deliver to Afghan villages everything asked of them by the [2006] COIN manual. The typical 21-year-old marine is hard pressed to win the heart and mind of his mother-in-law; can he really be expected to do the same with an ethnocentric Pashtun tribal elder?

Ultimately, direct, large-scale Western expeditionary counterinsurgency, especially in Afghanistan, has proven politically and financially unsustainable. In his 2014 memoir, Why We Lost: A General’s Inside Account of the Iraq and Afghanistan Wars, retired US Army Lieutenant-General Daniel Bolger has written of how Western intervention led to “unlimited irregular conflicts with limited forces”. Like Maechling and Eikenberry, Bolger regards expeditionary counterinsurgency as a heroic failure since its effectiveness is constrained by a deadly combination of incomplete Western military authority, inept host governments and the ticking clocks of “end dates” imposed by the West’s domestic electoral cycles. It is an iron rule that, in providing assistance to a government under siege from insurgents, no amount of foreign military skill can compensate for an absence of indigenous political will and local military capacity. The collapse of a large, well-equipped, US-trained Iraqi army at the hands of Islamic State fighters in 2014—less than three years after American military withdrawal—is a stark testament to this truth.

The failure of a population-centric counter-guerrilla campaign in Afghanistan and the near collapse of the Shia Iraqi state at the hands of Islamic State bring into perspective the third reason for Western counterinsurgency’s demise: the attempt to make counterinsurgency a strategy. Flushed by the success of the Iraqi surge of 2007, many COINdinistas argued that counterinsurgency as “an art by itself” could now be elevated into a fully-fledged “twenty-first-century strategy” for US statecraft. Such an approach was problematical from the outset since the enthusiasts seemed to forget that any strategy of warfare is bounded by the logic of political rationality and not by the grammar of military operations. As one expert, David H. Ucko, has pointed out, “counterinsurgency provides operationally oriented theory, not strategy”. Yet, after 2007, the idea of a “COIN strategy” achieved a powerful momentum in Washington and General Petraeus and his military circle became relentless advocates of a “comprehensive counterinsurgency approach” to achieve success not only in Iraq but also in Afghanistan.

The great flaw in promoting counterinsurgency as a strategy for America’s irregular conflicts was that it required the kind of political and military control associated with colonial rule. As the campaign in Afghanistan wore on, it became clear that much of the content of FM 3-24—as reframed by American neo-classical theorists—actually continued to reflect a twentieth-century colonial rather than a twenty-first-century expeditionary model of counterinsurgency. The ideas of Thompson and Galula proved misleading in the sense that both thinkers were indelibly shaped by “domestic” Cold War era models of counterinsurgency in Malaya and Algeria. In both of those countries, colonial regimes with national conscript armies led by military supremos such as Gerald Templer and Maurice Challe were in full control of local administration. In May 2007, Lieutenant General Sir David Richards, the British commander of ISAF, vividly highlighted the differences between colonial and expeditionary counterinsurgency in Malaya and Afghanistan:

[General] Templer in Malaya was running a single-nation effort and everyone beneath him would do as they were told. In my position [in Afghanistan] there were thirty-seven nations, all of whom could second-guess what I wanted; there was also a President [of Afghanistan], a number of ministries and the UN to satisfy. I could not have hoped to pull all the levers in the same way as Templer did; I needed to influence and needed to convince them as best I could.

The inability of the Western coalition to fully control the security environment in Afghanistan swiftly exposed counterinsurgency as little more than an ersatz strategy. Indeed, the lack of will and the venality of the Kabul government meant that counterterrorist-style “enemy-centric” attrition was far simpler and quicker for the Coalition to prosecute than pursuing “population-centric” counterinsurgency measures that required close and long-term Western-Afghan co-operation.

Counterterrorism “kill-capture” operations using drones and commandos against Al-Qaeda and the Taliban became common and contributed yet further to the decline of “comprehensive counterinsurgency”. As Michael Boyle demonstrates in an important March 2010 essay in the journal International Affairs, titled “Do Counterterrorism and Counterinsurgency Go Together?” there are significant differences between counterinsurgency and counterterrorism efforts in intervention operations waged by expeditionary forces. In an interventionist situation, counterinsurgency operations should ideally be designed to build the capacity of the host government. Yet the intractable political problem encountered by foreign forces is that if a host government is ineffective then building the regime’s local capacity only further enhances popular discontent. By contrast, if an intervention force pursues a counterterrorist approach—involving a quest for the targeted eradication of enemy insurgents and their supporters—then a host government may be required to expend whatever political capital it may possess by approving foreign actions that make it appear to be a puppet regime. In short, in an interventionist context, counterterrorism and counterinsurgency methods can easily work at cross-purposes and, if not balanced correctly, can result in deepening an insurgency. These kinds of strategic-level subtleties often eluded Western politicians. For example, in October 2009, David Miliband, the United Kingdom Foreign Secretary, betrayed a high-level ignorance when he declared, “we [in the British Government] believe in serious counterinsurgency. Counterinsurgency is a counterterrorist strategy.”

In the American context, one only has to peruse Bob Woodward’s account of the complex strategic deliberations of the Obama administration in his 2010 book Obama’s Wars to grasp the truth of Boyle’s argument. In the course of 2009, the Obama administration oscillated between two political strategies for improving a deteriorating security situation in Afghanistan. The first was a “counterinsurgency heavy approach” based on maximum population security measures and a long-term US commitment advocated by ISAF Commander, General Stanley McChrystal, and supported by the Petraeus circle and its various allies in the State Department and Pentagon. The second involved a restricted “counterterrorism plus approach” based on specific enemy-centric operations against the Al-Qaeda-Taliban franchise which was promoted by Vice-President Joe Biden and officials on the National Security Council. At the end of 2009, President Obama, keen to disengage from Afghanistan, decided upon a “counterinsurgency lite” compromise involving a further deployment of 30,000 troops to restore immediate security. However, the President’s “Afghan surge” was politically bounded in a manner that abandoned “nation-wide counterinsurgency”. US military action in Afghanistan was characterised as possessing only “counterinsurgency elements” and was carefully linked to a phased transition of responsibility for security to the Kabul regime by the end of 2014.

In what can now be seen as a turning point, comprehensive counterinsurgency was rejected by the Obama administration in December 2009 and replaced by a limited strategy to “disrupt, dismantle and defeat Al-Qaeda in Afghanistan and Pakistan and prevent its capacity to threaten America and our allies in the future”. This minimalist strategic approach only served to stimulate the rise of counterterrorist-style “kill-capture” operations by American forces. In May 2011, it was just such an operation that finally led to the elimination of Osama bin Laden in the now famous Zero Dark Thirty cross-border raid by US commandos into Pakistan. In retrospect, the death of Bin Laden can also be seen as the symbolic moment when the West’s experiment with large-scale counterinsurgency came to an end.

While is now clear that “the second counterinsurgency era” of 2004–14 has failed—and for many of the same reasons as the first era of 1961–73—it would be the height of intellectual irresponsibility to again banish the study of insurgency into the military wilderness. One of the deepest causes of Western failure over the past decade can be traced to the long neglect of irregular warfare. It was such neglect that led to the scramble for instant knowledge and equally instant experts between 2004 and 2007. At this time, there was such a dearth of organisational knowledge on irregular warfare available that policy-makers found it difficult to resist COINdinista proposals for interventionist counterinsurgency. As a former Vice-Chief of the US Army, General Jack Keane, reflected in April 2006:

We put an Army on the battlefield that I had been part of for 37 years. It doesn’t have any doctrine, nor was it educated or trained to deal with an insurgency … After the Vietnam War, we purged ourselves of everything that had to do with irregular warfare or insurgency because it had to do with how we lost that war. In hindsight, that was a bad decision.

In the absence of institutional counterinsurgency expertise, the intellectual vacuum created was filled by assorted nation-building romantics, faddish journalists and enthusiastic tacticians rather than by informed policy analysts, serious scholars and hard-headed strategists.

Given the continuing turmoil in the Muslim world and the multicultural composition of most Western societies it is almost certain that we will be confronted by irregular movements and Islamist non-state terrorist groups for years to come. We must, therefore, address the challenge of protracted insurgency through equally sustained and long-term analysis of the kinds of policies and interventions that are realistic.

Two long-term requirements are necessary. First, counterinsurgency must no longer be studied in pseudo-academic isolation, as it has so often been. The frequent banishing of counterinsurgency to the backwaters of strategic analysis turns the subject into a military version of “the doctrine of eternal recurrence” in which short periods of political necessity and military enthusiasm are followed by longer eras of strategic irrelevance and operational neglect. This cycle of “boom and bust” only divorces the study of counterinsurgency from the mainstream of professional war studies and encourages antiquarian scholarship and a focus on tactical methodology rather than on analytical history and strategic purpose. Without the latter approach, the linkage of counterinsurgency theory to practice inside a political framework can only be conceptually flawed. As a subject, counterinsurgency needs to be broadly situated, not narrowly isolated; it needs to become a component of a much wider study of irregular conflict devoted to a long-term and integrated analysis of intervention and expeditionary operations. In this manner, counterinsurgency will cease to masquerade as “an art within itself” and can be productively linked to other important fields in the irregular canon—such as counterterrorism, stabilisation, civil–military relations, and comparative cultural values in warfare.

Second, as we seek to re-invent counterinsurgency in the context of irregular conflict we need to avoid embracing a parallel illusion: namely that Western aerospace technology and special operations forces can bring about lasting strategic results. The Hollywood-style marriage of the aerial wizardry of Top Gun with the deadly commandos of Zero Dark Thirty may be satisfying to Western electorates weaned on television, but it is insufficient to counter the murderous ideology personified by a movement such as Islamic State in Iraq and Syria. In the future, military establishments and intelligence agencies need to consider a much more comprehensive repertoire of “under the radar” intervention skills in globalised conditions that are more modest, but at the same time, more sustainable over time. These intervention skills include forms of political warfare using strategic communications systems and proxy forces; security force assistance and specialist training teams; the use of counter-subversion and information operations and, where appropriate, a careful mixture of ground troops, clandestine special operations units and aerospace strikes.

Although the West cannot resolve the internal crisis of Islamic civilisation by direct military intervention, we can help shape events and outcomes by employing indirect military force. We must learn how to treat the irregular aspects of intervention operations with cumulative rather than sequential pressure, adopting a “limited liability” approach pursued over a long period of time. In this context, rather than focus so heavily on the lessons of Malaya, Algeria and Vietnam, more attention needs to be given to indirect American-led approaches that have had considerable long-term success such as Greece in the late 1940s, the Philippines in the 1950s and Central America in the 1980s. Comparative studies such as these highlight that Western intervention policy must be based on the fundamental need for effective governance by host regimes, and they help to indicate how military assistance might best be framed and calibrated. Any government under siege from insurgents must prove itself worthy of Western assistance by assuming the basic responsibility of sovereignty, namely to “own its fight”. For policy-makers, the issue is not what a threatened government ought to do, but what it can do. Local capacity and the triangular relationship between a host government, its population and an insurgent threat must be carefully analysed, with Western intervention based on matching military ways and means to strategic ends.

Conclusion

As we enter 2015, the USA and its allies have largely abandoned their twnety-first-century brand of anti-Islamist, large-scale counterinsurgency as a failed experiment in warfare. Paradoxically this decision has been taken at a time when the global tide of insurgency, particularly in the Islamic world, is rising, not ebbing. As a result, strategists, scholars and policy-makers must seek to re-invent counterinsurgency as a sophisticated, but politically bounded, component of irregular conflict.

Given the recent strategic frustrations of the American-led Western experience in Iraq and Afghanistan, counterinsurgency scholar D. Michael Shafer’s judgment on the efficacy of intervention against guerrillas in third countries continues to resonate as a historical warning. In his 1988 book, Deadly Paradigms: The Failure of US Counterinsurgency Policy, Shafer wrote, “two problems haunt American counterinsurgency … irrelevance and counterproductivity”. Irrelevance comes from the fact that foreign intervention cannot substitute for any host government’s political determination to fight for its own survival and, in so doing, to introduce necessary political reforms. Counter-productivity stems from the reality that, once Western forces move from assistance to assuming the burden of responsibility for combating insurgents, they risk putting on the poisoned shirt of Nessus as occupiers.

We should remember what the Taliban’s Mullah Omar said of General Pace’s “fast food” Americans in Afghanistan: “they have all the wristwatches; but we have all the time”. The electorates of Western countries clearly lack the patience, the will and the wallet to rebuild fragile states over decades. Yet, because the war between the West and radical political Islam is an existential clash of value systems that extends into multicultural liberal democracies, we cannot escape a long conflict. We need to counter a dedicated irregular enemy by adopting more indirect but effective uses of military power that enable us to be more judicious and far-sighted in our statecraft and strategy. All of this will take much toil, skill and a public willingness to endure in a twilight conflict fought on many fronts. This is a preferable course to our valiant but misguided large-scale counterinsurgency efforts over the past decade. After much cost in blood and treasure, the West has come to understand the mirage of armed nation-building in the Muslim world that T.E. Lawrence warned of in Seven Pillars of Wisdom:

We had been hopelessly labouring to plough waste lands; to make nationality grow in a place full of the certainty of God … Among the tribes our creed could be only like the desert grass—a beautiful swift seeming of spring; which, after a day’s heat, fell dusty.

Michael Evans is the General Sir Francis Hassett Chair of Military Studies at the Australian Defence College, Canberra, and a professor in the School of Humanities and Social Sciences, Deakin University. He was the lead author of the Australian Army’s December 2009 doctrine manual, Counterinsurgency, and currently co-ordinates an annual course in Irregular Conflict for senior Australian and international military professionals.