You can't play an RPG without monsters, and you can't play a horror RPG without some really, really f$%&ing scary monsters. Here are some of our favourites.
Excruciator (Monsters and Other Childish Things)
There are two things we can all agree are really scary: mean maths teachers and eldritch horrors from another dimension. Now imagine if both of those things were combined into one thing. That's the Excruciator. A mean, terrible maths teacher who is just a major bummer to your fourth-grade self, but occasionally she also turns into a formless void with tentacles that wants to drag you into the darkness forever.
Vampire Elder Brain (Ravenloft)
This is the Big Bad from the "Thoughts of Darkness" module for the 2nd Edition D&D Ravenloft setting, and it's deeply unsettling for a whole list of reasons. First, it's a giant tentacled brain that can read everyone's thoughts. It has an underling, an octopus-faced illithid, that wants to become a vampire so it can be more powerful.
But the only way it can do this is by implanting an illithid egg in a host body, then turning the host into a vampire before the egg hatches. Then he uses a weird machine to transfer his mind into that of the newborn baby illith-vampire, then grows up. He eventually latches onto the elder brain as a literally vampiric parasite, and the whole situation is just really gross.
Doll Master (Chill 2nd Edition)
Creepy dolls are a classic horror standby. I had a roommate once who collected dolls and I literally never watched tv in the living room the entire year I lived year because there were shelves of those damn dolls staring at you. But it took the Chill role-playing game to actually codify the creepy doll concept, give it stats, and clearly delineate how it was going to kill you.
The Doll Master is a spirit with the ability to infuse nearby dolls with its essence, animating them to do its bidding. So with one monster, you get an entire gaggle of murderous dolls.
Meenlock (D&D 3rd Edition)
These weirdos fall into the "sentient beings corrupted into awful little monsters" category, which is a really creepy kind of monster. Just look at them! You can sort of feel their anguish at being turned into something so dreadful. Yet they use their powers to fill your head with paranoia and fear, then drag to off to make you into a meenlock.
Ancient Stones (Night's Black Agents)
I know you're thinking, "How are ancient stones scary?" Well, Night's Black Agents is a game about fighting a worldwide vampire conspiracy, and it gives you the tools to create your own vampires, which can vary wildly in appearance and abilities (which keeps the players on their toes). These particular vampires are so bizarre and alien that we can't even perceive them correctly.
They exist in other dimensions, and manifest in our dimensions as giant statues. They have strange effects like warping causality around them. There's something about an alien intelligence so inscrutable we can't even fathom its motivations.
Swile (Chill 2nd Edition)
There's a really great 3rd Edition of Chill out right now, but part of the charm of the old edition is the art. It's... idiosyncratic, to say the least. And this monster, which is some kind of weird ice spirit, has a form that is a baby seal with a human face. This is deeply unsettling. Okay, yes, and kind of ridiculous. But can you imagine being out on an Arctic research mission and you run into a seal with a human face? No thanks.
Yibb-Tstll (Call of Cthulhu)
This space demon/god is originally from horror writer Brian Lumley's corner of the Cthulhu mythos, and it starts out terrifying enough: a grotesque head with eyes that flop around, while its blood manifests as an inky sentient blackness that drown victims. But the Call of Cthulhu RPG took things a step further: Yibb is usually hiding a bunch of nightgaunts under his wings, suckling at his eldritch teats.
His touch drives mortals insane, with one delightful detail — sometimes they are driven to rush toward Yibb-Tstll to suckle as well, shortly before dying spectacularly.
Mohrg (D&D 3rd Edition)
This is your basic undead creature, just a skeleton, really. But picture a skeleton that still has slimy intestines inside it, and a long, long tongue it lashes out at you. A long, slimy undead tongue.
Incidentally, if you'd like to check out some off-the-beaten-path horror RPGs, the current Bundle of Holding includes a whole handful of indie horror games like Dread, Kingdom of Nothing, Murderous Ghosts, and others.