The set-up: Characters are in a house, there is a axe-wielding psychopath on the looking to chop them all up. The characters hear a noise on the other side of the door.
Traditional RPG style of play:
Players draw their weapons, assume a defensive stance, cast buff spells, check their HP, etc.
Is this fun? Sure, it can be.
Is it scary?
I would argue no.
Sending an unbeatable foe against the PCs in a game where success or failure is very granular is easy to do. The party is 5th level, put them up against a 10th level threat. A 12the level threat.
Sure, that can be done.
But is it scary? Is it balanced? Is it fun?
As a GM if I want to stack the deck in my favor and “win” that is always easy to do.
But, horror gaming, gaming that can legitimately scare your players and their characters demands a different style of play.
As a GM, I can admit my shortcomings. Bringing a sense of doom to the table might not be one of them. This is not to say that I have not had success, but it can be difficult. There has to be buy-in from the players and GM, everyone has to commit to the conceits of horror gaming.
On those special occasions when I want to run a very special horror game, I tend to cheat.
I fall back on rules systems that force the players to be scared.
New-fangled RPG style of play:
Dread is an RPG that strips away pretty much everything that other RPGs have: stats, skills, etc, they do not matter. What matters in Dread is building tension, and that sense of impending death with any action.
How does it do it?
Build a Jenga block, any time a character is doing something potentially dangerous, the player must draw a Jenga block.
I ran a session of this, and added an on-the-fly house rule. When moments were building to a head, when time was of the essence, I instituted speed Jenga.
“Ok, you want to make it down to the docks in time to catch up to the ferry? I need 4 successful draws in one minute. Go!”
Jenga demands quiet.
It demands concentration.
And it brings with it a sense of… wait for it…. Dread.
Players might want to try and find an old AK-47 in Old Man Johnston’s house. Fine, let them. It doesn’t matter. It doesn’t increase their odds of survival.
Jared Sorenson’s Squeam does things a bit differently, the setup of Squeam is designed to emulate a horror film.
The coolest part of it is that, when the characters hear that noise on the other side of the door, one of them may very well open it.
Because that character failed their Curious roll.
And if that isn’t just like a horror movie, I don’t know what is.
Yes, it takes autonomy and free-will away from the players, but the end result is that it forces the players and their characters to do the things that make horror films fun!
I have run this several times for very-special episodes of my long-running campaign, and it is a blast. Darken the room, light a candle or two, appreciate the silence and darkness, and have a ton of jump-scare fun.
I tend to use a variant of the Scooby Doo rules set, and I could see the Cthulhu variant also being fun.
There are of course plenty of other games designed to scare, Spectrum Games (for whom I work) has Slasher Flick and Macabre Tales, the latter designed for 1-1 Lovecraftian play, the former designed for troupe style Slasher Flick play, and of course there is the grand-daddy of them all, Call of Cthulhu.
There are GMs out there who can make any system scary. My hat is off to them.
Me? I cheat.
I pick a game that has been designed to make the experience scary.