The Swedes, who are half-way through their season, started the match far sharper than the Scottish champions and secured a vital away goal through a wonder strike from Daniel Larsson in the 16th minute.
Malmö continued to have the better chances with Brazilian Wilton Figueiredo causing several panics in the Rangers defence.
Steven Naismith twice missed gilt edged chances for Rangers after the break but Allan McGregor was by far the busier keeper as Malmö take a vital advantage into next week’s second leg tie in Sweden.
“One of the worrying things from our point of view is we didn’t start the game at all well,” Rangers manager Ally McCoist said of his first European match in charge at Ibrox.
“But I do believe we created enough in the second-half to indicate that the tie is far from over. I really believe we can go over there and score so there’s all to play for.”
Rikard Norling felt his side had started the game well.
“We kept them back a bit by playing the ball well. That gave us a good start into the game,” the Malmö coach said.
“We played a very good side and it will be difficult for us at home as well. We will have to be at our best.”
At Ibrox, Rangers were given their first warning in the ninth minute when Figueiredo evaded two defenders in the box to sting the palms of McGregor with a ferocious strike with Jimmy Durmaz side footing the rebound wide.
Steven Whittaker crossed for Nikica Jelavic to head wide but disaster struck for the full-back at the other end when his mistake led to the opening goal in the16th minute.
The Rangers defender was robbed of possession at the edge of his box by Durmaz and Figueiredo backheeled the ball into the path of Larsson whose thunderous strike flew past McGregor for a crucial away goal.
Brazilian Figueiredo then pulled a shot wide before his deflected long-range free-kick went straight to McGregor as the home fans’ frustrations grew.
Captain David Weir had returned from suspension but only lasted until the half-hour mark when he hobbled off to be replaced by Ortiz. Rangers then finally tested Dusan Melicharek with the Czech keeper at full stretch to stop a curling free-kick from Steven Davis.
Rangers began to go forward with more purpose as the break approached with Madjid Bougherra hooking an effort wide and Jelavic could do little when a Davis cross deflected off him and out.
Malmö regrouped at the interval and caught the Rangers defence napping when Figueiredo raced down the right and sent in a cross that Larsson couldn’t connect properly with.
The Brazilian then tried to curl a free-kick from wide on the left into the top corner after Larsson had been felled by Sasa Papac but McGregor tipped it over.
Chances were few and far between for Rangers but Lee McCulloch tested Melicharek with a low drive from 25 yards that the keeper turned round the post.
The Rangers defence seemed to shudder every time Figueiredo broke forward and the playmaker used his skills to hold off the attentions of two Rangers defenders before releasing a shot that McGregor just managed to push over.
Naismith missed a golden chance in the 67th minute when Jelavic rolled the ball across goal from the right but the striker stretched to hit it with his right and poked it wide.
Just a minute later, in a carbon copy move, Jelavic again found Naismith at the far post but this time the Scotland international was off target with his left foot with the goal gaping.
Jelavic and Naismith then combined to set up Davis but he placed his shot from the edge of the box over the bar.
Substitute Dardan Rexhepi and Jilon Hamad then both came close as Malmö secured only their third away win of the season.