This article is more than
6 year oldDr Jackson was accused of questionable drug prescriptions, drunkenness and wrecking a government vehicle.
In a statement, he railed against the "baseless and anonymous attacks on my character and integrity".
Dr Jackson was set to have his Senate confirmation hearing on Wednesday, but it was postponed.
Senators from both parties said they wanted to examine allegations made by 23 colleagues and former colleagues of Dr Jackson.
The claims were outlined in a two-page document, details of which Democratic staff of the Senate Veterans' Affairs Committee briefed to US media.
According to the file, Dr Jackson was described as "flat-out unethical", "100% bad temper", "toxic", "explosive", "abusive" and "volatile".
He was also branded "the most unethical person I have ever worked with", "dishonest" and "despicable".
They also said Dr Jackson would "lose his mind over small things", and was "vindictive" and "belittling".
Speaking by telephone to Fox News on Thursday morning, US President Donald Trump said Dr Jackson was the victim of "false allegations, they're trying to destroy a man".
But Mr Trump said that he advised him that "a day or two ago I saw where this was going".
"The Democrats are obstructionists. It's horrible what they're doing," added Mr Trump, sounding agitated.
In a statement confirming his withdrawal, Dr Jackson said: "Going into this process, I expected tough questions about how to best care for our veterans, but I did not expect to have to dignify baseless and anonymous attacks on my character and integrity.
"The allegations against me are completely false and fabricated," said the 50-year old Iraq War veteran.
"If they had any merit, I would not have been selected, promoted and entrusted to serve in such a sensitive and important role as physician to three presidents over the past 12 years."
Dr Jackson has served as the White House physician since 2006, and rose in profile after declaring Mr Trump to be in "excellent" health following a medical examination.
He famously told reporters that Mr Trump could live to the age of 200 if his diet improved.
The White House revolving door: Who's gone?
Dr Jackson was selected by Mr Trump last month to lead the 360,000-employee agency, which commands an annual budget of $186bn (£133bn).
The previous Veterans Affairs Secretary, David Shulkin, was fired by Mr Trump last month following a critical report by an internal watchdog.
Mr Shulkin had been previously been unanimously confirmed by the Senate.
Newer articles
<p>The deployment of Kim Jong-un’s troops has added fuel to the growing fire in recent weeks. Now there are claims Vladimir Putin has put them to use.</p>