No, your book does not have to be set in New York City to be successful. There are bestselling books set in all sorts of locations all over this world and any other.
That said, for international success it does help if a book is set in a recognisable and well known location. Everybody knows where New York City or Paris or London is and has some kind of image, no matter how inaccurate, of that location in their head. Meanwhile, hardly anybody knows where Dogpoo, Nebraska, is (probably because I just made the town up) and what it looks like. Of course, you can still set your novel in Dogpoo, Nebraska, but you'll have to do more work to make your setting come to life in people's minds.
There is a big tradition of German regional crime fiction and there are crime novels (and movies and TV shows) set in almost every part of the country. However, the ones that get translated and find international success are set in Berlin, Berlin and more Berlin. Because international readers likely have no idea what East Frisia or the Allgäu or the Eifel, to take three popular locations for regional crime fiction, are like, but they have some kind of idea what Berlin is like.