Footage from the demonstration on Sunday, which is circulating on social media in Sweden, includes chants in Arabic which are understood to translate to "slaughter the Jews" and "stab soldiers".
Isaac Bachman, who has been Israel's Ambassador to Sweden since 2012, posted on Facebook on Monday that the messages coming from the southern Swedish city of Malmö were "very disturbing".
"These are extremely troubling instances of a grotesque but nevertheless very real – and murderous – incitement which must be dealt with by the full force of the law!" he wrote.
Meanwhile others were split on whether or not Swedish politicians Hillevi Larsson (from Prime Minister Stefan Löfven's Social Democrat party) and Daniel Sestrajcic (from the Left Party), should have spoken at the rally.
The Palestinian Culture Society (Palestinska kulturföreningen), which arranged the demonstration, told Swedish tabloid Expressen on Tuesday that the event had been organised to "help Palestine" and said that it had been "individuals" who had talked about knives and not anyone representing the group.
There has been a surge in violence in in Jerusalem and the West Bank in recent days including a number of attacks by young Palestinians on Jewish targets. Israeli police shot dead two men wielding knives on Saturday, while a female police officer killed a Palestinian woman who stabbed her in Hebron.
Palestinians have accused Israeli officials of being too quick to shoot, while there are claims that knives have been planted at the scenes of some of the attacks.
Relations between Israel and Sweden remain tense after the Nordic nation became the first in the EU to formally recognise the state of Palestine a year ago.
Bachman was briefly recalled to the Middle East from Stockholm but returned a month later.
In mid-December Israeli media reported that Israel's Foreign Minister Avigdor Lieberman had shunned the chance to meet his Swedish counterpart Margot Wallström, who ended up postponing a visit to his country.
Wallström however cited "diary reasons" as the cause of her decision.