The EU-Ukraine Association agreement, which was strongly criticized by Russia, is designed to deepen political and economic integration between the EU and Ukraine by allowing free trade across borders.
But the nationalist Sweden Democrat party's first two MEPs, Kristina Winberg and Peter Lundgren, who were both elected in May, were among those who voted against the idea.
MEPs backed the agreement in Strasbourg on Tuesday with 535 votes in favour, 127 against and 35 abstentions.
The European Parliament's President, Martin Schulz described the deal as an "historic moment". It takes Ukraine one step closer to joining the the EU in future.
MEP Peter Lundgren told The Local:
"Since we are a Eurosceptic party, we don't want to increase the European Union, because our citizens are paying for that without their direct permission".
"The European Union is aggravating an already sensitive situation", he added.
He argues that the agreement "was rushed through committee without proper consideration or usual procedures".
The news comes as the Sweden Democrats celebrate scoring almost 13 percent of the vote in Sweden's general election on Sunday, a result that has rattled other political groups on all sides of the political spectrum.
The Sweden Democrats want to leave the EU and campaigned on a platform to dramatically cut immigration to Sweden. No other party in Sweden is openly against putting limits on the number of people moving to the country from abroad.
As Sweden's incoming Prime Minister Stefan Löfven seeks to form a coalition government, his Social Democrat party is concerned that the Sweden Democrats will try to block some of its key proposals from being passed in the national parliament.