It depends on what aspect of a place makes me want to return to it.

I completely agree. I don't love Madrid for the same reason I love Ho Chi Minh City. Wandering around Paris is not the same as wandering around Phnom Penh. Each have their appeal and their drawbacks. I've been to very few places that I wouldn't like to visit again.

Sometimes I find myself craving an experience from a place I've visited. For example, I had a filetto alla griglia in Florence one time that made any food I ate for several days seem like it wasn't worth eating. To me, it was the epitome of what the experience of enjoying a steak should be. Likewise, I remember the rainy season in Vietnam. I'd look out the windows of my room and watch as walls of water advanced over the ocean, betting against my housekeeper about how many minutes it would be until the rain lashed the windows, how the rain was the only thing that quieted the constant flow of traffic on the street below and the sudden noise of heavy rain on a corrugated iron roof.

For me (and I suspect many others) traveling is less about the place and more about the experiences one has had.