"I'm a bigger liability for my party board than an asset," Westling told the TT news agency.
He explained that he considered his future with the party on Friday, finally concluding to step down from his political responsibilities.
The footage which prompted Westling's decision has been published over the course of the week by the Expressen newspaper.
The scandal, dubbed SD-Gate by some in the press, has so far caused economic policy spokesman Almqvist to leave his post and justice policy spokesman Ekeroth to take a "break" and continues to dominate the Swedish media.
In addition, the incident has exposed a rift within the party, with some factions criticizing the way party leader Jimmie Åkesson has handled the crisis, with some experts arguing the incident could cause the party to split.
On Friday, Westling admitted he and the others in the film had acted inappropriately.
"I regret that I didn't see what was happening; that I didn't break it up sooner, that we didn't get out of there," he said in reference to the events captured on the controversial video.
"We behaved incorrectly, you can't behave like that. It's clear one has to accept the consequences for their actions; that's part of life as an adult."
Westling added, however, that he planned to remain an "active member" of the Sweden Democrats. While he doesn't have a seat in parliament, he has served as an alternate board member for the party in Stockholm as well as a member of the Sweden Democrats' foreign policy committee.
In a recent post on his own blog, Westling wrote about the situation for homeless people in Stockholm.
"Stockholm can't take responsibility for other country's homeless," he wrote in October.
Westling, 29, describes himself on the opinion website Newsmill as Catholic, nationalist, and an EU opponent.
Story continues below…
A poll conducted by Novus for Sweden's TV4 in the wake of the scandal has found that 40 percent of Swedes have less confidence in the Sweden Democrats, while 33 percent said they had less confidence in Åkesson.
However, 64 percent of respondents who identified themselves as Sweden Democrat voters said the racist video scandal hasn't affected their view of the party, and 25 percent said the incident had actually increased their confidence for the Sweden Democrat leader and the party.