The two men, Mohamoud Jama, 23, and 26-year-old Bille Ilias Mohamed, had been sentenced in December by the Gothenburg district court to four years in prison.
Swedish prosecutors appealed the verdict, only to have the Court of Appeal for Western Sweden release the two men at the conclusion of the trial, pending the announcement of the verdict.
In a unanimous ruling, the appeals court came to the same conclusion as a minority of justices on the district court, namely that the evidence was insufficient to convict the men.
"The appeals court therefore dismisses the case against both people," the court said in a statement
The court ruled that prosecutors hadn't been able to back up their claims, which meant that the court never examined the human rights aspects of the case concerning whether or not the al-Shabaab militia was involved in an armed conflict.
The men had been held in remand for several months in a case in which prosecutors relied on evidence gathered from wiretaps carried out by Swedish security service Säpo.
The original lower court decision had said the two Swedish citizens "had taken it upon themselves, and decided with the Somali Islamist al-Shabaab militia to commit terrorist crimes in the form of suicide attacks."
It said both men were members and sympathisers of the al-Shabaab movement, which has declared allegiance to Osama bin Laden's al-Qaeda network and controls most of southern and central Somalia.
Both men denied the charges against them, but one of them has admitted that he spent time in an al-Shabaab training camp.
The appeals court however said the evidence presented by prosecutor Agnetha Hilding Qvarnström was not enough to convict the two.
Tapped phone conversations showed that both Jama and Mohamed "have strong sympathies for the terrorist organization al-Shabaab and the organization's methods and that they ... went to Somalia with the intention to physically join al-Shabaab."
It was shown that "they were in fact in contact with al-Shabaab and ... expressed that they were not opposed to or even wanted to die as martyrs," the appeal court said in its ruling.
However, it pointed out, "it is not punishable ... to have sympathies with terrorist organizations or even to join an organization, if joining does not automatically entail a commitment to commit a (terrorist) crime.
"It is also not punishable to have decided to commit a terrorist crime as long as the personal conviction does not lead to a decision made with another person," the court said.
It could not be ruled out that the two men aimed to have another role within al-Shabaab than that of suicide bombers, the court added.
Jama's lawyer Richard Backenroth said the freeing of his client proved the Swedish judiciary worked well.
"It's a happy end to this story that the prosecutor and security police have been working on," he told the local Göteborgs-Posten (GP) daily.
Prosecutor Qvarnström said Wednesday she was not surprised at the ruling, pointing out that the court had decided at the end of the trial three weeks ago to release the two men pending the verdict.
"This was not completely unexpected," she told the TT news agency, adding that the case had been "very difficult" to prosecute.
Asked whether she planned to appeal to the Supreme Court, she said it might be difficult to get the country's highest court to review a case where the jurisprudence was not in question, but which had been dismissed due to a lack of evidence.