Adm Jackson called a string of allegations published by Democrats, including that he had crashed a government car while driving drunk, “completely false and fabricated”.
He said that “going into this process, I expected tough questions about how to best care for our veterans, but I did not expect to have to dignify baseless and anonymous attacks on my character and integrity”.
In a statement the former combat medic said he was “regretfully” walking away from the nomination to lead the US government’s second-largest agency, which oversees healthcare and benefits for some 20 million military veterans.
Democrats had claimed in a dossier published on the day of his confirmation hearing that Adm Jackson had been drunk on duty and gained the nickname “Candyman” for his prescribing practices. The hearing was postponed.
The allegations were based on interviews with nearly two dozen former and current colleagues of the White House physician.
Mr Trump appeared to suggest Adm Jackson should withdraw earlier this week, saying the confirmation process had become “too dirty”.
Adm Jackson said when withdrawing from nomination on Thursday that it had become a “distraction” for the president.
In his statement, he added: “The allegations against me are completely false and fabricated.
“If they had any merit, I would not have been selected, promoted and entrusted to serve in such a sensitive and important role as physician to three presidents over the past 12 years.
“In my role as a doctor, I have tirelessly worked to provide excellent care for all my patients. In doing so, I have always adhered to the highest ethical standards.
“Unfortunately, because of how Washington works, these false allegations have become a distraction for this president and the important issue we must be addressing – how we give the best care to our nation’s heroes.”
Democratic senator Jon Tester admitted not all the allegations against Adm Jackson had been verified.
He told MSNBC: “Am I 100 per cent rock-solid sure that he did this? No, but I’ve seen a pattern here that continues on and on and on.”
But his decision to “flaunt” the unverified document was described as “outrageous” by White House legislative director Marc Short, who suggested Mr Tester was using the allegations for political gain.
Speaking on Fox News, Mr Trump called the allegations “false”.
Shortly after Adm Jackson dropped out, the president called into the Fox & Friends morning programme to praise the doctor as an “incredible man” who “runs a fantastic operation”.
He said he has a new candidate for the post with “political capabilities” but did not announce who it was.
CNN, citing an unnamed official, reported that Adm Jackson would remain as White House doctor.