Yusupha Sallah, 32, and his toddler Yunus were attacked on a bridge in Malmö by a gang who racially abused him and threatened to kill his son.
"They called me 'you black bastard' and such, and said they would kill me and my son," Sallah told the Expressen newspaper.
The incident is being treated as a hate crime by police. Sallah added that the attack unfolded after one of the members of the gang, which at first was one man and three women, kicked a toy out of his son's hand and he tried to intervene.
"What can you do? The man pushed my son and when I picked him up, he hit me on the back of my neck. We fought. The women pulled at my shirt while the man beat me up. As I was trying to protect myself, another ten men came running towards me and they beat and kicked me," Sallah told the Sydsvenskan newspaper.
The gang then attempted to throw Sallah off the bridge, which has a five-metre drop, but he managed to cling onto the other side of the barrier while his son was screaming.
"I thank God that I was able to hold on as I could have died if I fell. They took off my shoes and hit my hands hoping I would let go. When they heard the police coming they stopped," the 32-year-old told Expressen.
A witness rescued Sallah's son Yunus and the father was later brought to hospital. Some of his teeth had come loose and he had multiple cuts and bruises. Sallah was released on Monday morning.
The native of Gambia has only been living in Malmö for eight months after moving across the water with his Danish partner Sabina Frederiksen. He moved to Sweden because of Denmark's rules over residency permits, and intends to remain.
"We have only been treated well in Sweden and are very happy here. This is the first negative thing that has happened," said Sallah.
"I'm not going to start hating anybody. All I hope is that the people who did this are punished and learn something from it."
The crime is currently classified as assault and battery but may be upgraded to attempted murder depending on witness statements. Police are still appealing for witnesses to come forward. At present nobody has been arrested for the attack.