"Perhaps it's been hard for us to accept that in the middle of our open, democratic society there are people sympathizing with the Isil (Isis) killers," he told journalists at a press conference in Stockholm on Thursday afternoon.
The Social Democrat leader added that among the Swedes travelling to Syria and Iraq to fight for Isis there could be people who are also a threat to residents of the Nordic nation.
"It's completely unacceptable that people can take part in terrorism and return [to Sweden] without being held responsible," he continued.
However the Prime Minister suggested that his country was ill-prepared for the growth of Islamist extremism in the Nordics.
"Sweden has been naive, both previous [centre-right] and Social Democrat governments."
Security has already been increased nationwide after Sweden was put on "high" terror alert for the first time on Wednesday as Swedish police searched for a man accused of "planning a terrorist attack" in the country.
Löfven said that his centre-left coalition government was looking at fresh ways to monitor extremist activities, noting that new technologies such as Skype had made traditional surveillance methods such as phone tapping more or less obsolete.
He suggested that there would be increased secret data and camera surveillance as well as a greater focus on using biometric passport data.
The Prime Minister also said that his centre-left goverment would work to tackle poverty and unemployment as part of efforts to discourage vulnerable young people from joining radical groups.
"Society should also send a clear signal giving youths hope for the future," he told reporters.
Asked how he felt when he first heard that a suspect was wanted by Swedish police for planning terror crimes in Sweden, he said:
"I was dismayed that we have people who sympathize with and want to fight for these evil powers. But we mustn't be naive (...) we know they exist. That's bad."
Sweden' Security Service, Säpo, revealed last month that around 300 Swedish nationals are believed to have left the country to travel to fight with Islamic extremist groups in Syria and Iraq.