I very much enjoyed Hilary Mantel's first two on Thomas Cromwell, and she won the prize for both Wolf Hall and Bring Up the Bodies. (Haven't read the third, and am sorry she is no longer with us to write something else.) She did fit much of the Booker description, though, except for being a female.
I don't entirely believe in a hard separation of genre from literary fiction. Wolf Hall could easily be described as historical fiction, which used to be a major genre--albeit, a mainstream major genre. When a strictly genre writer gets really successful, bookstores take them off the genre shelves and put them with the literary and mainstream fiction. I've seen it happen with all genres.
However, I have zero aspirations to be mainstream or literary. My characters have substantial vocabularies but are otherwise down to earth. I don't let them quote W.H. Auden--or Byron for that matter.