Two quickfire first-half goals gave the Swedish minnows hope of a sensational comeback at the Emirates, but Sead Kolasinac's goal just after the interval saw Arsenal hang on for a 4-2 win on aggregate.
The north London giants move into Friday's last-16 draw, but Östersund will take all the plaudits for a brave display full of quality.
“At 2-0 we thought we had a chance. We played well, won the game but it was just short of the miracle,” Östersund coach Graham Potter told BT Sport.
Having cruised to a 3-0 victory in the first leg last week, Arsene Wenger's Arsenal appeared complacent and were caught out by the vibrant visitors to the Emirates.
Midfielder Hosam Aiesh capped off a slick move for the underdogs to pull one back in the 22nd minute, running onto Saman Ghoddos' through ball and slotting into the corner with the aid of a deflection off Calum Chambers.
The hapless Arsenal centre-back was caught out just 69 seconds later as the home fans were left stunned, with winger Ken Sema spinning Chambers and drilling a low finish past Gunners goalkeeper David Ospina.
Potter may have been thinking back to his side's missed injury-time penalty in the first-leg 3-0 loss, but they continued to press as Arsenal laboured.
The Premier League side made it to half-time with a 3-2 aggregate lead intact, but were booed off the field by their own supporters.
Unsurprisingly, Wenger's men were much-improved after the break and left-back Kolasinac capitalised on uncertainty in the Ostersund defence to hammer home with his weaker right foot.
But instead of building on that goal, Arsenal continued to be largely outplayed by their unheralded opposition, with Ghoddos and Curtis Edwards both going close to restoring the two-goal cushion.
The Swedes ran out of legs as their continental adventure came to an end with a famous victory, but Arsenal were left facing plenty of questions ahead of Sunday's League Cup final against Manchester City.