The girl's parents have appealed the decision, Metro reports.
"We talked about lots of names and then Elvis popped up. We thought it was a name that was both pretty and gender-neutral. We're not Elvis Presley fans at all," the infant's mother, Linda, told the newspaper.
The most important factor for the couple, who do not own any of The King's records, was for the name to be "gender-neutral".
But not everyone agreed with their assessment that Elvis fit the bill. In its ruling, the tax board wrote:
"It is the National Tax Board's view that Elvis is a first name of a masculine type and as such may, in light of standard practice, be considered clearly inappropriate as a first name for a woman."
The girl's mother however believes that her appeal will be successful and has vowed to continue calling her daughter Elvis whatever the eventual outcome.
The family can also take heart from the experience last year of the Tomaro family, whose daughter was eventually allowed to keep the name Metallica after a protracted battle with the authorities.