Television images showed a small group of Pegida demonstrators in a cordoned-off area of the southern city's central square, Stortorget, surrounded by anti-racism protesters.
"There are at least 3,000, most of them counter-demonstrators," Malmö police spokesman Lars Förstell told news agency AFP, adding that there were about 30 Pegida protesters.
"The support is much greater than that," Pegida's Swedish leader Henrik Rönnquist told broadcaster SVT, putting the number at 140.
"There is an unmet need for this in Sweden. It's not about racism, it's not about shutting out immigrants. It's about our values and traditions," he said.
Police on the ground put the number of counter-demonstrators closer to 5,000, according to local newspaper Sydsvenskan. Pegida attempts to deliver speeches were drowned out with chants of "no racists on our streets". No serious incidents were reported, but police were forced to intervene to extinguish smoke bombs and flares hurled at the Pegida side of the square.
But counter-demonstrator Sandra Jonsson told broadcaster SVT at the scene on Monday evening: "I am here to show that this kind of thing should not happen in our streets. This is certainly not okay and I'm fuming."
Five speeches had been planned for the rally, but little was heard through the noise of the crowd.
"They don't have any speakers, so it's impossible to hear what they're saying. But that's nothing we can do anything about," said Ewa-Gun Westford of the Swedish regional police.
Malmö is Sweden's third city and home to a growing population of Muslims. The Swedish Church said they had received threats after announcing plans to hold a pro-Islam service during the march.
Pegida (Patriotic Europeans Against the Islamization of the Occident) rallies began in the German city of Dresden last year with several hundred
supporters and snowballed to reach 25,000 people on January 12.
But numbers have fallen since the movement's founder stepped down on January 21 after a picture surfaced of him posing as Adolf Hitler. Other senior figures have also since resigned.
Small offshoots of Pegida have sprung up in other German cities and marches have taken place in the Czech Republic, Austria, Denmark and Norway, involving however only a few hundred people and generally outnumbered by far larger anti-racism rallies.