The study, done by researchers at Uppsala University and Karolinska Institute in Stockholm, examined information provided by around 106,000 men and women in the country.
It showed that those who consume a large amount of milk run the risk of an earlier death than those who don't due to increased chronic low-grade inflammation in the body. And the new research is the first to suggest a difference in risk according to gender.
"In a previous study we observed that a high level of milk consumption is linked to an earlier death. However this new study shows that there is indeed a gender difference, which is something that hasn't been shown before," Uppsala University's Karl Michaëlsson told The Local.
"My advice though is to see this study as a piece of a puzzle. We need more pieces of the puzzle before authorities can give definitive recommendations," he added.
The study, published in the American Journal of Epidemiology, followed up on previous research which suggests that milk drinkers are not better protected against bone fractures (for women, the opposite was actually observed).
The new research suggests that regular milk drinkers do however risk shortening their lives, with the risk at its highest among women who drink a lot of milk.
"That could be explained by the gender differences in how women and men break down galactose, a component of the milk sugar lactose. Among animals there are clear gender differences in this area," Michaëlsson noted.
For women who drink at least three glasses of milk and eat fruit and vegetables a maximum of one time per day, the risk of dying earlier is almost three times higher than for women who drink no more than one glass of milk per day and eat fruit and vegetables at least five times per day.
Women who drink three glasses of milk per day and eat fruit and vegetables at least five times per day still showed a 60 percent higher risk of earlier death than women who consumed the same amount of fruit and veg but drink little to no milk at all.
For men it is a different story however. The risk of early death is only 30 percent higher for men who drink at least three glasses of milk per day than it is for men who rarely or never drink milk. In contrast to women, the amount of fruit and vegetables they consume did not appear to significantly alter those outcomes.
Asked if the study shows that women in particular should moderate their milk consumption, researcher Michaëlsson took a cautious stance.
"The study is an observational study and it alone should not be used as a basis for recommendations. We need more pieces of the puzzle."