Mr Trump and his co-defendants had alleged Fani Willis's relationship with Nathan Wade, who she hired, had compromised the integrity of the case.
The judge disagreed - but said it did create an "appearance of impropriety".
He said either Ms Willis or Mr Wade should leave the case to resolve that.
In his ruling, Judge McAfee said Ms Willis had committed an "tremendous lapse in judgement" by engaging in an affair with Mr Wade, and also called her testimony last month "unprofessional".
Mr Trump and his 18 co-defendants claimed the relationship created a conflict of interest, saying Ms Willis benefited financially from the relationship.
But Ms Willis and Mr Wade denied this - saying there was no financial benefits, for example they split the cost of their holidays together.
Judge Scott McAfee said there was not sufficient evidence that there was an actual conflict of interest.
But he found there had been an "appearance of impropriety" - and there was a "need to make proportional efforts to cure it" before the case can continue.
In his 23-page ruling on Friday, Mr McAfee presented Ms Willis with two options: to step down, along with her team, and have the Prosecuting Attorney's Council take the case over, or have Mr Wade step down and allow "the public to move forward without his presence or remuneration distracting from and potentially compromising the merits of this case".
"[A]n outsider could reasonably think that the district attorney is not exercising her independent professional judgment totally free of any compromising influences," Mr McAfee wrote. "As long as Wade remains on the case, this unnecessary perception will persist."
Mr McAfee added: "[A]n odor of mendacity remains".
Donald Trump and 18 others are being prosecuted in Georgia for conspiracy to overturn the state's 2020 election results - which they deny.
Earlier this year, Fulton County District Attorney Ms Willis - who brought the case - admitted she had a romantic relationship with Mr Wade, but said it had no bearing on the case.
Mr Trump and his co-defendants alleged that this relationship created a financial conflict of interest, saying the couple used the money paid to Mr Wade to fund luxury trips together.
Newer articles
<p>The deployment of Kim Jong-un’s troops has added fuel to the growing fire in recent weeks. Now there are claims Vladimir Putin has put them to use.</p>