Reacting to comments from EU Justice Commissioner Vera Jouriva, who gave the latest estimate to French newspaper Le Figaro on Monday, a spokesperson for Säpo told The Local that it remained difficult to monitor those who were making the journey from Scandinavia to the Middle East.
"About 50 of those known to have travelled to Syria have returned to Sweden. They travel to Syria, return to Sweden and tend to go back again," said press officer Fredrik Milder.
Säpo confirmed that at least 150 Swedish residents were known to have been to Syria or Iraq to fight for Isis or other extremist groups, with intelligence suggesting that at least 35 had died in the process.
But Milder added that "there is only very little we can do to stop people from travelling from Sweden to join al-Qaeda inspired groups".
EU Justice Commissioner Vera Jouriva told Le Figaro that the true total number of fighters from across was likely to be far higher than even the latest estimate of 5000 to 6000 due to the difficulty of tracking foreign fighters in the conflict.
"At the time of the attacks in Paris and Copenhagen, we decided not to allow ourselves to be guided by fear," she added, referring to January's twin Islamist attacks in the French capital and the subsequent deadly shootings on a cultural centre in Denmark.
Focusing on those seeking to leave for Syria to wage jihad, or those returning from the conflict, meant intervening "too late", she said, reflecting Säpo's later comments.
Jouriva said the EU instead wanted to promote prevention as a means of curtailing the steady flow of European nationals, looking at the diverse reasons of why people joined jihadist groups beyond simply religion.
British research had identified "a desire for adventure, boredom, dissatisfaction with their situation in life or a lack of prospects," in those who had opted to leave their families behind and head for Syria, the commissioner explained.
Another focus for the EU was speeding up the exchange of information between the police forces and court systems of member states, she said, with more intelligence sharing required. Sweden's Home Affairs Minister Anders Ygren is among several top European politicans who have been promoting this approach in recent months.
"On the EU level it is about collecting information about the journeys of terrorists and better cooperation between security services to help exchange information about specific events," he told The Local in January.
Säpo has told The Local that some individuals who are feared to be preparing to travel from Sweden to Syria to fight are interviewed by officials.
"Based on available intelligence, the service also investigates people potentially involved in the recruitment of others to commit acts of terrorism in Sweden or abroad," said Milder.
Isis or the Islamic State, which rules a swathe of territory in Syria and Iraq is also believed to be attracting growing numbers of foreign fighters from outside Western Europe.
More than 215,000 people have been killed in Syria's four-year war, which is increasingly dominated by jihadist groups.