Changing of the Guard

British Prime Minister David Cameron, who leaves office on Wednesday after barely six years in power, racked up political enemies at an impressive rate during his tenure.

The main Labour opposition were furious at the fierce austerity measures he introduced in efforts to guide the country out of the financial crisis, yet were frustrated that, even with such unpopular policies, ‘Teflon Dave’ was unbeatable at the ballot box.

The Liberal Democrats, with whom he formed a coalition government in 2010, grew frustrated by what they viewed as Cameron’s pandering to his socially conservative right-wing and he then ruthlessly targeted their seats at the 2015 general election, reducing their M.P.s from 56 to eight.

And in his own Conservative Party, many lawmakers were irritated that their leader rarely listened to, let alone reflected, their views, such as their hatred of the U.K.’s membership of the European Union or their opposition to gay marriage.

Tributes have, of course, been flocking in from Cameron’s supporters before he hands the keys to Number 10 Downing Street to his successor, Theresa May. Cameron’s health secretary, Jeremy Hunt, said there is “huge gratitude for what he’s achieved for the country”, while culture secretary John Whittingdale praised his “extraordinary service” to the country.

Read more: The 6 Moments That Defined David Cameron’s Leadership of the British Government

But those opponents believe the Cameron years have left behind a more divided country, exacerbated by his decision to hold a referendum on E.U. membership – and his failure to convince the electorate that it was in their best interests to remain in the bloc. TIME has spoken to a range of Cameron’s political rivals to issue their verdict on the Cameron era:

The Labour Party

Angela Eagle, who is currently standing to be Labour leader, was subjected to one of Cameron’s most infamous – some would say sexist – Parliamentary barbs in 2011, when he mimicked a popular TV commercial to urge her to “calm down, dear.”

Eagle highlights a common assertion that Cameron was arrogant, with too much faith in his powers of persuasion. This hubris, it is argued, led him to call a risky E.U. referendum the country was not demanding, simply to placate some of his own lawmakers who had campaigned for years to leave. “He ought to be remembered for kowtowing to the forces of Euroskepticism in the Conservative Party,” she says, “for doing his homework at the last minute, and accidentally taking us out of the E.U.”

The former shadow business secretary also turns on Cameron’s economic record. Although he has been praised for record high levels of employment, the Office for National Statistics figures show the poorest 10% earn nearly 10 times less than the richest 10%. “He has left us as one of the most unequal advanced economies and has decimated communities up and down the country in the most unfair way,” she says

The Scottish National Party

The SNP grew significantly during the Cameron era, having swept Scotland in last year’s general election, winning 56 out of 59 seats against only six in 2010. But the left-leaning party says the government’s austerity measures have done irreparable damage to the country.

Pete Wishart, the S.N.P.’s shadow leader of the House of Commons, tells TIME Cameron has “a toxic legacy”, arguing the overwhelming historical fact of his premiership will be the “calamitous decision” on the E.U. But Wishart also criticizes Cameron’s government for missing a series of economic targets.

He is also furious about the ‘bedroom tax’, which meant people in housing programmes lost part of their benefits if they had a spare bedroom. “”This will never be forgiven nor forgotten,” says Wishart.

Cameron’s premiership, he says, will rank even lower than that of Tony Blair, now a much-derided figure in the U.K. because of his support of the U.S. invasion of Iraq. “It’s a Hobson’s choice,” says Wishart, who does praise Blair for giving Scotland greater autonomy. “The total failure of Cameron’s Government means I edge just a bit towards Cameron [as being worse].”

The Liberal Democrats

While many give Cameron credit for introducing gay marriage legislation in 2013 in defiance of the far right of his party, Liberal Democrat Norman Lamb argues that his party were the driving force behind this law. But, he adds: “He agreed to it and saw it through, quite bravely really, because he lost a lot of party members over it.”

Lamb, who served as a health minister during the Coalition years, also argues that Cameron was an effective manager, given he held together the first postwar coalition Government for a full five years. Lamb says: “In time, the Coalition Government will be seen as a well-functioning Government. He made necessary compromises – in 2014 he acceded to my demands for more money for the National Health Service – and maintained good relationships at the heart of Government.”

But, he says, Cameron failed to present a coherent vision for the country while in office. “I’ve no idea what Cameron stands for really.”

The Conservative Party

Within Cameron’s own party, Steve Baker led the charge to get the U.K. out of the E.U. Baker was highly critical of Cameron’s attempts to renegotiate the terms of the country’s membership of the bloc, memorably condemning the results as “polishing poo” this year.

Baker says Cameron had a strong record on public service reform, particularly on education where schools were given greater freedoms, and that he at least granted the U.K. a referendum, arguing this should have been held in the early 1990s. “Of course, Cameron will be remembered for giving us a referendum – and being on the wrong side of it.”

That’s not the most damning verdict, but it is the most telling.