I'd say whether (and how much) to pare down is a question of what will make you happy.
Over the years, I've pared down by a small number of books, but I've kept the vast bulk of them. I'm sure interior decorators would cringe, but every empty wall space in the house has its book case. Most of them have two layers of books. I got some wonderful real wood ones that have space for three layers. About 11 years ago, I made my patio (which I really didn't use) into a room, partly for general storage, but the walls are all lined with book cases.
Aside from being an avid reader, I think I have the collector's gene. My parents were both collectors, in particular my mom, who ended up with a huge glass collection. I've been a book collector from when I was young. By the time I was in college, I could do research papers without leaving home. (This is before the internet was really a thing.)
My continuous expansion slowed considerably with the advent of ebooks. Now I buy in ebook format unless there isn't one available.
My reading tastes are eclectic, and I never know what I might want to read or reread. Fantasy, science fiction, horror, literary fiction, literary criticism, history, biography, politics, theology, comparative religion, mythology, art--all are well-represented. I probably could get rid of the education books I accumulated while I was teaching, but the reality is those wouldn't be big sellers for a thrift shop. I am about to get rid of the out-of-date books on software and other aspects of computer technology. (So I'm not completely irrational about books.)
All of that said, if I found staring at the books depressing, I'd pare down more. I find it calming instead. Their presence isn't hurting me in any way, so you know what? They're going to stay. I also think about the good times I had (in the old days when book stores and used book stores were more plentiful) accumulating the collection. A lot of times, I can pick up a book and remember which long-gone book store it came from and how I felt when I first bought it. I also think about how hard many of them would be to replace. I don't think I have anything a rare book collector would faint over, but there are certainly a lot that are long out-of-print and essentially irreplaceable. (I think the oldest one is an 1868 campaign biography of Horatio Seymour, but I have a fair number from the late Nineteenth and early Twentieth Centuries. But even much more recent ones are often out-of-print.
There's one final consideration. Two of my heirs are avid readers. I'm only 62, so that's a far-off consideration, and I'm sure they're not going to be interested in everything, but it's hard to say which ones they might want.
Of course, that's me. If your books make you on balance more unhappy than happy, then by all means pare as much as feels comfortable. On the other hand, as Vijaya says, "If you have space and enjoy having them, what's the harm?"
