I think given the evidence of the time, Malthus had sound reasoning. What he didn't (probably couldn't) predict was the stunning growth in technology that has increased crop yields by orders of magnitude, as well as the rise of the middle class, with its de-emphasis on having a large family. Organisms choosing not to breed given the opportunity is pretty new. But you're definitely correct that when a hypothesis is failing, its the hypothesis, and not the evidence that needs to be tweaked. That's a big problem in social science generally, I think. What's the old joke in economics? Something like, "Those data are interesting, but do they fit the theory?!"
Great point cgod. It is surprising to me that Malthusian ideas are still in vogue (I even encounter them among graduate students in my department)! I think it has to do with the fact that humans seem to default to a dystopian narrative of the future. This is the common line of reasoning: If x, y, and z factor remain the same (e.g., agricultural output, standard of living, etc.), and all of our current problems remain the same (e.g., climate change, increasing population size), then catastrophe will ensue. Of course under this line of reasoning catastrophe will always be on the horizon (hence persistent Malthusians). And b_b I couldn't agree more. As I stated in the article, Malthus was not a bad scientist. He simply could not have predicted what the future of the human population would be, there was not enough data and something like the global emergence of our species has never happened before! Also, I have developed the opinion that we do not actually have a carrying capacity (so ecologists can stop looking for one). I think that our carrying capacity is flexible and dependent on our technological capabilities. What do you guys think about that?
Carrying capacity is a system parameter that can only be determined empirically, as far as I know. It is a function of many things itself (e.g. food supply, disease prevalence, predation, and--in the case of humans--new technologies for energy extraction), and it is definitely not static. I'm sure we'll asymptotically approach some population, but I don't think that this population is necessarily our carrying capacity, at least it won't be if we can find a cheap, reliable, clean source of energy.