Much of Ulvaeus's wealth from his Abba days, as well as from his own music production, has been held in an endowment on the Isle of Man, off the west coast of Britain, reported Dagens Nyheter.
Royalty payments and income from record sales have been channelled through a network of trusts in Holland and other tax havens, ending up in the endowment managed by the Royal Skandia insurance company.
In the 1970s the Abba members signed a number of agreements which transferred their rights to various companies.
But the Swedish Tax Board now says that agreement was a sham, that he was operating from Sweden and that the royalties should therefore have been taxed.
In 1984 Ulvaeus moved to England but since then has lived for many years in Sweden. According to the tax board's calculations, almost 100 million kronor in royalties were paid through the network of foreign companies between 1998 and 2003.
In 2003 Ulvaeus declared Swedish income of 2 million kronor.
"The tax board's basis for the decision is that Björn did not sign over his rights, but they were transferred according to the rules, and not because of the tax effects," said Björn Ulvaeus's legal advisor, Sven Rygard, to Dagens Nyheter.
"It was for English reasons that it was done when Björn moved there in 1984. That's what the tax board has now found fault with. Obviously we're going to appeal against this. We're pretty convinced that we'll be proven correct here."
The Swedish Tax Board's demand includes increased tax assessment, penalty charges and interest, totalling 86.9 million kronor.