Coach Pia Sundhage's side went behind 2-0 to Dzenifer Marozsan's sweet strike and Lisa Sembrant's comical own goal before Stina Blackstenius gave Sweden a lifeline.
With Brazil's men taking on Germany in their final tomorrow, the boisterous home crowd roared on the Swedes in the final stages.
However, it was to no avail as Sundhage missed out on a third consecutive Olympic title after leading the USA to gold in 2008 and 2012.
“After 90 minutes the feeling is a little bit empty, but I said to the players after that we have won a silver medal rather than losing the gold,” said Sundhage.
“I am so proud not just of the result, but also the way we played today against one of the best teams in the world,” the coach continued.
“We were so close. It got very emotional,” striker Lotta Schelin told Dagens Nyheter after the match.
Along with their silver medals, the team can console themselves with the fact that they have made history in Rio as the first ever Swedish women's football team to claim an Olympic medal.
Sweden had been criticised, most notably by outspoken USA goalkeeper Hope Solo, for their defensive tactics in upsetting the world champions and Brazil in the previous two rounds on penalty shootouts.
However, they started on the front foot against the two-time world champions, against whom Sweden had lost their previous nine championship matches.
The team in yellow in blue should have gone in front when Olivia Schough spooned over Lotta Schelin's low cross.
However, eight-time European champions Germany took control of the game at the start of the second half with a goal worthy of a cup final. Dzsenifer Maroszan gathered a cross ball on the edge of the box and had time to steady herself before curling a perfect strike into the top right hand corner of the net.
They extended their lead 14 minutes later thanks to an unfortunate own goal. A Sara Daebritz free kick from 20 yards smacked the post and Linda Sembrandt tried to clear but instead kneed the ball into her own goal from just a few yards out.
Despite Blackstenius' goal from Olivia Schough's beautiful low cross reducing the deficit shortly after, a late Swedish rally failed to produce an equaliser.