The pair were out fishing off shore when the man stumbled and fell into the water at around 4pm on Tuesday.
The 11-year-old boy was able to hold on to his granddad for a several hours and as the boat approached a harbour he could no longer hold on.
The boy was then forced to watch as his granddad was pulled out to sea by the strong tide.
The boy managed to steer the boat into Bua harbour and called for help at around 6pm. When he reached safety the boy was suffering from severe shock and was taken into the care of relatives, the local Hallands Nyheter reported.
The Swedish sea rescue service (Sjöräddningen) launched a search and rescue operation on Tuesday evening but were unable to locate the man despite the deployment of divers and a helicopter.
The man was not wearing a life-jacket and the chances of finding him are considered slight.
"We are doing all we can, but it doesn't look good," said Annika Vestergård at Sjöräddningen.