It's kind of better that way, IMO. A bunch of people come look at the place, and most discover it's a decent place but not really for them. But several find it's exactly what they want, and they are a benefit to the rest of us in my experience. Over the last three years this has probably happened roughly 10 times, and every time the site is annoying for like two days, but turns out to be better in the long run.
Yeah, ButterflyEffect and I had a talk about this via PM the last time it happened, with the Reddit Conspiracy thing...man, what a crap-show that was. I don't think anyone from that migration stayed that time around. I'm down with anybody who appreciates high-effort content to stick around, though - two or three high quality people as opposed to an extra-large user-base of not-so-high-quality people, hehe.