"Anti-prostitution work needs to focus even more on men's role," according to the report entitled Men and Equality.
The report was compiled for Sweden's Minister for Gender Equality Maria Arnholm by a group of experts, including academics, an entrepreneur and a trade unionist.
Sweden was the first country in the world in 1999 to make it illegal to pay for sex but legal to work as a prostitute.
However, despite thousands of fines -- up to 7,500 kronor ($1,164) in some cases -- no one has been imprisoned for the crime as provided for under the law.
Although Sweden's sex purchase law is supported by a majority of Swedes, the authors of the report say more needs to be done to target men who continue to pay for sex.
"First of all it's about bringing attention to men who buy sex. Men's attitudes to paying for sex need to be discussed more, despite the difficulties in finding out about those who pay for these services," they wrote.
Studies carried out on men who have sought help to alter their sexual behaviour show, according to the report, that "a large proportion" of them "have problems with destructive or compulsive sexuality".
"Today there are initiatives in Stockholm, Malmö and Gothenburg aimed at men who pay for sex and which work to support men with changing their behaviour," they added.
Their recommendation to do more to prevent men going to prostitutes contrasts with Prime Minister Fredrik Reinfeldt's call for a tougher approach, including the enforcement of prison sentences.
In December he announced that his party, the centre-right Moderates would campaign for minimal jail sentences for those found guilty of knowingly paying a minor, or someone who has been trafficked, for sex.