After learning of the relationship, the woman's angry husband turned to the parish vicar and the bishop, both of whom viewed the affair as inappropriate, reported Sveriges Radio.
The woman's husband also confronted the priest directly on several occasions before the two finally came to blows in October 2005.
Police gave up their investigation of the incident after being unable to determine which man threw the first punch.
Then in December 2005 the priest received a letter from the parish notifying him that he'd been fired because his behaviour and actions had done serious damage to the church's reputation.
But the priest felt that he'd been wrongly relieved of his duties and in August of 2006 sued his former employer for wrongful termination.
The Västmanland District Court first ruled in favour of the priest, citing the length of time that had elapsed between the inappropriate behaviour and the parish's decision to fire him.
In reaffirming the district court's decision, the Labour Court ruled the priest was entitled to lost wages as well as 100,000 kronor ($16,700) in compensation for the insult of having lost his job.