"It would be difficult to have someone so responsible for the fiscal crisis in the UK at the helm of the IMF," Borg said of Brown in an interview with UK broadcaster Sky News.
He went on to say that "a country with a 10 percent deficit is a little bit problematic" as a place to find a successor" to Strauss-Kahn, adding that Brown was "not that strong" a candidate to head the IMF.
Later on Thursday, Borg however praised the strengths of Lagarde as a possible replacement for the disgraced Strauss-Kahn.
While he said it may be too early to take a formal position, he had been very impressed by Lagarde, seen as a key member of the current French government.
"One person whose name has come up is that of French finance minister Christine Lagarde and I must say that she has made a very good impression on me," Borg told Sveriges Radio (SR).
"She has been a real driving force in (the European Union finance ministers group) and she has been very important in the work of the Group of 20 – she has the influence and the experience," he said.
Borg also noted that the possibility of having a woman at the IMF helm was another positive point for Lagarde.
Earlier, Swedish foreign minister Carl Bildt said that Strauss-Kahn, who resigned on Thursday as head of the IMF amid sexual assault charges, had been a great asset to the world during the global financial crisis,
"It should be said that DSK served the world with great distinction during a most critical period," Bildt said on his official Twitter account, stressing that finding the right "successor will be important."
Borg also noted that Europe, the biggest contributor to IMF funds, should again claim the top job, as developing countries including China voiced hopes that someone from the emerging economies be considered.
Europe has traditionally supplied the head of the IMF while the United States names the World Bank chief, the two bodies set up after World War II to help guide the global economy.