The pensioner, who is a patient at the hospital in Kalmar, got a hold of two kitchen knives and began making stabbing motions with them. He later pressed a knife against the chest of another patient whom he was sharing a ward with reports the local Barometern newspaper.
Staff at the hospital attempted to calm the man down by keeping him in a room before four police officers arrived on the scene
He tried to throw knives at the police who used shields and ducked for cover before pepper spray was used to bring the situation under control.
"He was going to throw knives so in order for the police to enter they must protect themselves," said Eva Jirhall of the Kalmar police to the TT news agency.
An eyewitness told the Barometern newspaper that the whole incident lasted two and a half hours and was really "nasty."
"It could have ended up much worse," said the witness.
The 95-year-old is now being cared for by the health service and is suspected of a range of offences including assault and vandalism.
The man who was attacked by the pensioner by pressing a knife into his chest is understood to be making a recovery.