Ulf Kristersson's Moderates party is part of a centre-right bloc called Alliance which received marginally fewer votes on September 9th than the previous governing centre-left coalition.
Both blocs have struggled to form a majority after refusing to make a deal with the anti-immigration Sweden Democrats, a party which was linked for years to far-right groups.
Kristersson's announcement came after some allies criticised his suggestion to build a minority government with the centre-right Christian Democrats party, but not other members of Alliance.
“I have tried all the other options to form a government and it hasn't succeeded,” Kristersson told a news conference in Stockholm.
Kristersson's failure to form a government comes after outgoing prime minister Stefan Löfven was ousted in a vote of no confidence last month.
Löfven's Social Democrats party — which largely dominated Swedish elections for decades — had governed for four years in a coalition with the Greens, and was supported in parliament by the Left Party.
The September election focused on the state of housing, healthcare and welfare services, in particular due to pressures from immigration following the 2015 migrant crisis.
The parliament's speaker Andreas Norlén — who has four chances of tasking a candidate to form a government which parliament will accept now has to find a new candidate.
He said he would hold a new round of talks with party leaders on Monday, making it likely for Lofven to get the next chance.
If all four attempts fail then Sweden will head to another election.
“The way I see it there are only two options. Either a party is willing to talk to us (to form a government) or we will head for another election,” Sweden Democrats leader Jimmie Åkesson told TT news agency.