Yeah, that was kinda my initial worry, although for $5 you're beating a CD price four times over, and the experience five times over, I also don't want to cherish something that's actually shit :D I'll be careful flea markets and thrift stores are your friends for cheap records. The only thing is to make sure they're not in awful condition.
Oh god, did I ever tell anyone on Hubski about the time where I thought I was the proud new owner of the White Album and then it turned out, I was only the proud new owner of about .75 of the White Album? But - I paid $2 for it - would rather half one playable record and one not for $2 than not at all. Thrift, pabst, thrift and gift. Buy really, really special albums you love and can't live without online but don't rush into it. I would say you should buy the first 100 good albums you can find in thrift shops. Prices range of course, usually between $.50 and $2/album, but I figure you could average out at about $1/per. Be patient. Don't let it burn a hole in your pocket, it's so rewarding to gradually build a collection.
If they're that cheap I guess I don't even care if the quality isn't good-as-new. Then again, the first records I heard were my dad's $250 box set of Radiohead's whole discography, played on a $10,000 sound system which creates instant transcendence, so it might be a bit different. Since I lack the proper unnecessary amount of cash to throw at shit, I will stick to your plan. Thrift and gift it is.