The driver reportedly refused to pick up passengers who looked foreign in a number of incidents in the Lapland town of Kiruna, which are now being investigated by police.
Transport official Magdalena Wäppling confirmed one incident in which the driver asked three people to get off the bus "because of the colour of their skin."
"The bus driver has left the company… We demand that companies providing local services respect our values — every person must be treated equally," she told news agency AFP.
According to a local television news report, the driver, who has since between sacked, repeatedly refused entry to minorities and failed to stop for foreign-looking passengers at snowy bus stops around the Arctic town on several occasions.
Temperatures in the town regularly drop below -20C in winter.
A passenger named Samson told public broadcaster Sveriges Television how the bus driver had pointed at him and ordered him off the vehicle, apparently because he was black.
"I didn't feel good. Even now, talking about it, I don't feel good," he said with tears in his eyes.
Another passenger, who wished to remain anonymous, said the driver told him to "get off the bus" and when he asked why she replied, "Because you are a black man."
The reports, which have received a great deal of attention in the Swedish media, come amid fears of growing xenophobia in the country.
Ironically Kiruna is often seen as an example of successful integration of newly arrived refugees with 52 percent in work, helped by the fact that the town of 20,000 people is built on top of the world's largest underground iron ore mine.
The anti-immigration Sweden Democrats party won 13 percent of the vote in September elections and became Sweden's third largest parliamentary party.
It holds the balance of power in local authorities in much of southern Sweden but has less support in northern areas which are threatened by depopulation and labour shortages.
It was the only party to campaign for Sweden to limit its intake of refugees amid record numbers of applications, expected to reach 80,000 to 105,000 in 2015.
The numbers have been boosted in the last year due to a pledge by the government to offer permanent residency to all Syrian refugees that arrive in Sweden.