The incident occurred while a debate between party leaders took place at the nearby Riksdag.
“The four people have been arrested and we are speaking with them just now. There were two on the roof but we've taken them down,” Stockholm police spokesperson Towe Hägg told The Local.
Initial reports suggested that two people had forced their way past security guards and into the building but a Rosenbad spokeswoman said this had not been the case.
TT also spoke to one of individuals involved in the incident, who explained that it was a protest against Swedish state-owned energy giant Vattenfall's deal to sell brown coal operations in Germany to Czech operator EPH.
The activists that climbed onto the roof of the Prime Minister's offices unfurled a banner with the message “Let the coal lie”.
Police commander Peter Tomasson said police were treating the incident as “very serious” after the protesters managed to erect ladders and hoist themselves onto the roof.
“It's a protected building and the security guards are armed, and there's a reason for that.”