I enjoy cons, but I mostly go to meet people, attend and take part in programming. IMO, cons are great for interacting with fans and fellow writers and for making connections. I do bring some books along, but so far I've never rented a table at a con.
I do have some experience manning sales tables from my time as the sales manager of the university magazine. We regularly set up our table at university events, literature events or simply in the cafeteria and we sold a lot more magazines than we would otherwise have. And indeed, the mag folded not long after I left.
Here's a few things I have learned: Decoration-wise, at least have a nice tablecloth. Ours was a bright green bedsheet with the logo of our magazine. It's also always good to have something to give away to customers and passers by. Candy or chocolate is always good. Actually, even if you don't have a sales table, bringing candy and chocolate to a con is still a good idea. I bought a bunch of German chocolate and candy to WorldCon 75 in Helsinki and gave them away to people I met. They'll remember you more easily if you give them something nice.
Swag is also good to have. Bookmarks never did much for us. But at university magazine, we had button pins with the logo of the magazine and/or inspirational quotes. We sold them for 1 Euro, I think (because we had to pay for them and had almost no budget), though you can also give them away. They were very popular, at any rate. Badge ribbons are also a popular bit of swag. You can order them and print your book or series title on the ribbon. And because people stick the ribbons to their badge, it's basically free advertising.
Here is a photo of my beribboned WorldCon 75 badge. If you just sit passively behind the table, you'll likely be ignored. So try to strike up conversations with people who pass by. If you have the temperament for it (many pople don't), you can also try the barker or market crier routine. We had one guy who was really good at it and it increased sales.
Finally, get yourself a lockable cashbox and fill it with sufficient change, because inevitably the first person will only have a fifty dollar bill or so. Also look into credit card reader solutions.