Almost hidden from outsiders, the US is engaged in a new war in the Middle East which is growing in intensity and running out of American control. Al-Qaida in the Arabian Peninsula (AQAP) has hugely extended its reach across southern Yemen in recent months after driving government forces out of several towns. "For the first time in history al-Qaida controls territory," an Arab diplomat in Sana'a, Yemen's capital, told me. "A year ago they were numbered in the dozens, armed with light weapons and scattered here and there. Now they are in their thousands with tanks and heavy weapons."
The movement's new armaments come from over-running government troops and bases. In Jaar, a town in Abyan, a province that borders the port city of Aden, AQAP has set up a police force and administers justice after the official police fled. AQAP has also sent teams to assassinate officials or kidnap and kill foreigners across the country. Saudi Arabia's deputy consul was abducted in Aden last month. The local security director in Hadramaut was seized a fortnight ago. Foreigners have been targeted in western Yemen far from the semi-desert regions of the south where AQAP started.
Security sources say AQAP fighters include Saudis, Somalis and Pakistanis with prior experience in Afghanistan. But AQAP's expansion also rests on its ability to recruit local people. "They have been clever. They have used local grievances, and not only a jihadi discourse," said the Arab diplomat.
Two decades after traditionally independent south Yemen was united with the north, separatist feelings in the south are on the rise. Southerners say little development money has come into the region while northerners are favoured in getting permits to run businesses or buy land in the south. The secessionist movement, Hirak, which has been active for several years, is gaining new ground, while AQAP seeks to compete by using the same complaints of marginalisation.
The Yemeni media in Sana'a rely largely on government sources for information about the hidden war. They claim the army is fighting back well but the number of places where clashes with AQAP are reported is widening in scope. Most significantly, officials in Washington have disclosed that the Obama administration is planning to increase the drone attacks on Yemeni targets, which it conducts from secret locations in the region. The CIA has reportedly sought permission from Yemen's new president, Abed Rabbo Mansour Hadi, to allow drone strikes even when targets have not been clearly linked to AQAP.
If this change in tactics is made, the US will be following the same disastrous model that has failed in north-west Pakistan, where the killing of civilians by drones has alienated the central government as well as local people and helped al-Qaida to recruit more supporters.
Some observers blame AQAP's surge on last year's political upheaval, which led the previous president, Ali Abdullah Saleh, to leave office. While the outbreak of widespread protest and massive street demonstrations in several Yemeni cities before Saleh left has allowed all kinds of social, economic and political grievances to emerge, and encouraged new groups to press for change, it would be folly to try to suppress them.
AQAP's increased activity is only one element in Yemen's spectrum of revolt. As with any insurgency, the best response is not foreign or local military intervention but to address the injustices and poverty which lie at its root. In March, the UN's World Food Programme reported that levels of food insecurity in Yemen had doubled since 2009, with 45% of the country's people short of food. Yet only 15% of WFP's previous appeal was funded. Far better for the US to fill that aid gap than waste more taxpayers' dollars on counterproductive drones.
Jonathan Steele is a Guardian columnist, roving foreign correspondent and author.