"This comes up in every board meeting we have," Financial Sector Union chair Ulrika Boëthius told business daily Dagens Industri (DI).
"They're held to task for something they have no power to change."
Around 90 percent of their members say customers are increasingly annoyed at them.
The same number of bankers said they thought criticism in the media was the reason behind the attitude shift.
The newspaper used the term 'bank bashing' and claimed that Finance Minister Anders Borg now takes every chance he gets to hammer home the message to bank customers to be more demanding.
Neither was financial Markets Minister Peter Norman spared from criticism.
"Our members get upset when he compares bankers to vacuum cleaner vendors," Boëthius told DI.