IMHO it largely has to do with the fact that her novels construct what seems to be a rational philosophy, but in actuality, propose a suggested way of living that doesn't stand up to logical scrutiny that would put it on par with philosophies that elucidate consistent and compelling perspectives.

Personally speaking, the antogonists (like Ellsworth Toohey) in her novels are bumbling caricatures of people, and the opposing views they represent are presented in the form of strawmen arguments. Conversly, her protagonists are very romanticized individuals with few complexities and even fewer flaws. In short, Ayn Rand didn't successfully present Objectivism as a serious philosophy, and didn't make the arguments for it in a serious manner. There's more selling than convincing there. The Fountainhead is a pretty good read, though. I think it makes for an effective self-help book.

Part of the problem with Objectivism IMHO, is that it creates a false duality of human behavior and then builds an argument upon this assumption. Rand pushes that selfishness is virtuous, whereas altruism is evil. However, it doesn't take much contemplation to realize that all human interaction is comprised of actions that can best be described to have selfish and altruistic components. Furthermore, some apparently altruistic behavior can be done for selfish reasons, whereas the opposite is also true. Objectivism is simplistic, and it is difficult to apply an 'objective' logical structure to problems in any consistent fashion. You have to assume initial conditions that requires over-simplification of problems. In fact, I'd say that Objectivism in practice is really Objectivism in-the-eye-of-the-beholder, which is actually a very subjective application.