What's the character's lineage?
I know you said opening scene, but you could have him remember something where his mother said something like "You look just like your father" and the way she said it made him feel horrible, because his father was a wife beater, adulterer, something, but just from the tone, the way she looked at him, or someone else did, it made an impression on him.
Maybe his mother always favored a sibling, there are plenty of ways to make a kid feel like crap and it will take a while for an adult to realize his looks create favorable first impressions when he knows deep inside he is the awful person he always was told he was growing up.
So something like "Hey handsome," or "Buy a girl a drink" or "Twenty pence to go upstairs" (I don't know the phrases for the time period obviously) and he is shocked that anyone would say that to him of all people.
Edit - Okay, my mind took off on this.
Joe entered the tavern. He was thirsty from the long ride and a tall glass of mead was the only thing on his mind.
"Hey Handsome."
Joe heard the words but ignored them. He was busy scanning the room for bar. Then he felt a sharp poke in his shoulder and looked down.
"You raised in a barn or something, I said hi handsome."
"Um," Joe said.
"You want a drink? This way."
She took his hand and he let himself be led across the room.
It wasn't until after he had swallowed a good portion of his drink that he allowed himself to think about it. In the orphanage (I made him an orphan) he was most often called "Ugly Bastard". All the orphans were bastards, obviously, but he was always "Ugly Bastard". When wench settled into his lap and leaned into kiss his cheek in he understood. He had heard about women like this.
"Half price for you gorgeous," she said.
Joe had just spent his last pence on the mead.
"I've no more money," he said.
"We'll work something out."
"Like what?"
"You can steal me a Starbucks tomorrow."
