Prime Minister Stefan Löfven invited the centre-right opposition Alliance to talks aimed at a “cross-bloc cooperation”, after his Social Democrats remained the biggest party with 28.4 percent of votes, its weakest election score in a century.
He said the election result marked “the death of bloc politics” in Sweden.
At the head of one of the few left-wing governments in Europe, Löfven's bloc appeared to hold 144 of 349 seats in parliament, one seat more than the Alliance, with votes in 99.8 percent of districts counted.
That is well short of the 175 needed for a majority.
Just 30,000 votes separated the two blocs. But some 200,000 votes from Swedes who live abroad, which could tip the balance, were only to be counted Wednesday.
READ ALSO: The Local’s live election-day blog
The four-party Alliance — made up of the conservative Moderates, Christian Democrats, Liberals and Centre — rejected Löfven's invite, urging him to resign as it reiterated its determination to form its own government.
In Sweden, the speaker of parliament typically consults all party leaders after an election before tasking the one most likely to succeed at forming a government.
Meanwhile, the far-right Sweden Democrats made steady gains, rising from 12.9 percent in 2014 to 17.6 percent.
Voters' frustration over immigration after the country welcomed almost 400,000 asylum seekers since 2012 has been closely connected to the growth of the populist party, while studies have found links between Sweden Democrat voters’ economic situations and dissatisfaction with traditional parties.
“We have strengthened our role as kingmaker… We are going to gain real influence over Swedish politics,” Sweden Democrats leader Jimmie Åkesson told cheering supporters at an election night party.
The Sweden Democrats remain the third-biggest party — failing to overtake the Moderates — and were credited with 17.6 percent of the vote, below the 20 to 30 percent Åkesson had hoped to win.
Marine Le Pen of France's far-right National Rally — formerly known as the National Front — hailed the Swedish party's rise, tweeting: “Yet another bad night ahead for the European Union. The democratic revolution in Europe is moving forward!”
Löfven had called the election a “referendum on the future of the welfare state”, but the far right presented it as a vote on immigrants and their integration.
The Sweden Democrats, with roots in the neo-Nazi movement, have said the large number of asylum-seekers presents a threat to Swedish culture and claim they put a strain on the country's generous welfare state.
Around 18.5 percent of Sweden's population of 10 million was born abroad, according to Statistics Sweden.
Löfven had urged Swedes not to vote for what he called a “racist party” as he cast his ballot Sunday.
“It's… about decency, about a decent democracy. And the Social Democrats and a Social Democratic-led government is a guarantee for not letting the Sweden Democrats extremist party, racist party, get any influence.”
The Social Democrats have led a minority government with the Greens since 2014, with the informal support of the ex-communist Left Party to pass legislation.
Mattias, a Stockholm resident at an election night party in the city, said he was “extremely concerned” about the far right's steady climb since it entered parliament in 2006 with 5.7 percent.
“The election is between potential democracy and potential fascism,” he told AFP.
The composition of the next government may not be known for weeks.
Lengthy negotiations will be needed to build a majority, or at least a minority that won't be toppled by the opposite side.
Speaking to supporters late Sunday, Moderate party leader Ulf Kristersson said he planned to build a government that would “unite our country and take responsibility”.
But in order to secure a centre-right majority in parliament, Kristersson would have to put an end to the Sweden Democrats' pariah status and open negotiations with them.
That could prove fatal for the Alliance, with the Liberal and Centre parties repeatedly ruling out a deal with the far-right.
Åkesson has said he would demand a curbing of immigration policy in exchange for his support, or key positions on parliamentary committees that draft legislation.
“When the same party time and again increases, and the other parties stand still, then you have to listen to that part of the population that is voting for this party.
“It's time to take responsibility and talk to the Sweden Democrats,” Sweden Democrats parliamentary group leader Mattias Karlsson told public broadcaster SVT.