I only started buying vinyl a couple of years ago, but I already have a crate full. I agree with Bfx though, the best albums were the best finds. Here's what I'd try to find:

The Wall by Pink Floyd, or Wish You Were Here. Original The Wall's don't have The Wall written on the front side. DSotM is great, but I've never liked Breathe nor Money much. The Wall is a musical story, not in the High-School-Musical kind of way but in the there's-more-depth-in-this-than-some-literature kind of way. If you can get Animals for cheap it's also a great album.

The Red Album / The Blue Album by the Beatles. They're somewhat chronological, so if you like early Beatles (Love Me Do etc) go for red and if you like later stuff go for blue. Both great though, which is why I own the two of them.

Toto IV by Toto. Has the well known songs (Rosanna, Africa) but the rest of the album is pretty solid too.

Bridge over Troubled Water by Simon and Garfunkel. I have this Greatest Hits which was like 2 euro in a bin somewhere, it has better songs but it feels kinda like I'm missing the songs in the context of the album.

Paris (Live) or Even In The Quietest Moments by Supertramp. Former is an amazing live concert on vinyl and the latter is my favourite album of theirs. Breakfast in America is more of a vinyl classic, Crime of the Century is great too but for me nothing beats EITQM. Only real flaw on the album is Babaji. Hate that song. But it's the first of side two so I just start side two with From Now On.

Chronicle by Creedence Clearwater Revival. Technically a greatest hits album but fuck it's just too good.

Now that you mention it, I don't even have Led Zeppelin on vinyl. But I prefer IV - When The Levee Breaks is such a fucking great song. Here, go listen.