Every successful species has one or more successful environmental adaptations. Tools given to them by nature with which they can control their environment. Speed, agility, or size are some examples.

Humans on the other hand are blessed with incredible intellect. Our problem solving skills are (from what I know) unmatched in nature. So much so, that we stand a good chance of destroying our planet with it.

What is scary, you ask. The removal of those very tools which allow you a greater degree of control. Because of this, anything unknown, foreign, incomprehensible or strange is scary to us. We need knowledge, in order to have control. Remove knowledge, and you introduce chaos.

I would wager a guess that to make a successful horror game, which doesn't implement the typical dark and tacky theme, you would need to find a way to present something vexing. Then remove the player's ability to understand it. Sure it could pose a challenge or a threat, even. But ensure that the player does not know what it is.

Take a look at the aliens in Metro 2033. Sure they can kill you, but once you understand them, they aren't frightening. Even though that doesn't dampen their lethality. Or watch the movie "Cube." Sure there's lots of gory killing involved, but once the story unfolds and the mystery is solved, the film stops being scary.