"Bilda has made a mistake, and we want to apologise for this," said Magnus Stenberg, Middle East coordinator at the association, in a statement on Friday.
The drawing by two Swedish pastors depicted three mice, one of which carried a weapon. The mice appear to be standing on a map of the Palestinian territories in a scene which could be interpreted to represent Israelis.
While the image had not yet been published in the exhibition, angry responses had been registered by the US-based Simon Wiesenthal Center which argued that it depicted Jews as rats.
"Animalization (depicting humans as animals) of Jews was perfected by the Nazi propaganda machine, an all-too effective way to dehumanize Jewish citizens in the eyes of their German neighbours," the centre wrote in a statement.
Magnus Stenberg however explained that the artists behind the work intended their image to "describe how all sides in a conflict are losers", a conclusion that they came to after visiting with Israeli and Palestinian artists during a trip to the region.
On behalf of Bilda he expressed regret that the drawing had been circulated and explained that the artists had become unhappy over the response via social media channels and had decided to withdraw their work from the exhibition.
"We should have realized how wrong a picture of cheese and mice could lead people's thoughts, and how, for example, could be exploited in anti-Semitic circles," Stenberg said.
Far-right website Nationell.nu was among those which celebrated the drawing, claiming that it showed that "Swedish artists grasped that Jews were violent pests".
The drawing was due to have featured in connection with Stefan Sjöblom and Larz Lindqvist's art exhibition "The holy land - det håliga landet" (land of holes) at the Immanuel International Church which opened on Friday and runs until April 10th.