It’s a lot of fun pulling the rug out from under your audience. Come on, most of us love that kind of thing; it’s one of the reasons why certain movies, TV shows and stories are so popular. Hell, M. Night Shyamalan has done his damnedest to build his directing career on it, with wildly varying results. It used to be a lot more popular in fiction than it is today, but there are still lots of examples out there, particularly in genre fiction.
Still, the twist ending has its problems. A lot of times, it can come off as contrived, especially if you force the twist onto the story, instead of letting it rise from the narrative. Anybody who saw the remake of Planet of the Apes a few years back knows what I mean; that ending, a pathetic attempt to match the “holy shit” factor of the original, was created solely to provide the frisson of surprise. Instead, what those poor suckers who paid full price to see it got was the ghastly stench of stupid. It didn’t make sense, completely contradicted the story elements, and overall was just dumber than a bowl of hair. There are many, many examples in literature of said dumber-than-hair endings, depressingly many. Let’s not dwell on those.
If, however, the story lends itself to a nice twist, and the writer can pull it off without cheating the reader, then by all means twist away. Such a construction is especially rewarding if the writer can sink clues as to the story’s eventual conclusion into the story, so that hindsight reveals them and readers smack themselves at not seeing it coming. Theodore Sturgeon was good at this kind of thing, and Richard Matheson still is. Cornell Woolrich was also good at this type of plotting; in fact, Woolrich was so good at it, he got tired of twist endings and started putting his twists in the middle.
Of course, one of the best practitioners of this type of plotting was one of the first to make it popular in America: O. Henry. O. Henry wasn’t the first to throw in twist endings (Guy de Maupassant had him beat, among others), but Henry made them a success, usually because Henry’s were funnier and more optimistic (of course, de Maupassant was French, so that could have been a factor). His most famous stories, such as “The Ransom of Red Chief” and “The Gift of the Magi,” used irony and humor to pull the mileage out of his stories, but more importantly, Henry allowed the characters and situations to guide the flow, so when the twist came, it fit right in. Henry did this kind of thing well, so much so that such twists are often called “O. Henry endings.” Some people say this phrase with a sneer, but O. Henry stories are still read today, and there’s even a respected literary award named for him, so those people can take a long walk off a short pier.
So, if you’re going to include a twist ending in your story (perhaps thinking of an old Twilight Zone episode or The Sixth Sense as you imagine, with evil glee, the look on your readers’ faces as you spring the trap), there are a couple of things you’ll want to make sure you have. To say you need a good twist is obvious, but before you can have that, you need a story that will support it. Thus, you need to have a plot arc, characters upon which said twist can be perpetrated, and an ending that makes sense. Please, please, please do not just throw one in because you think it will be nifty: you will end up looking stupid at worst, not playing fair at best.
Most importantly, the twist has to arise organically from went before. You have to lay the groundwork in the story, using the elements mentioned above, and you have to let the story work its way to the twist. It can be ironic and funny, as in “The Ransom of Red Chief,” where (ATTENTION: SPOILER ALERT) the kidnappers find that they have to pay a ransom to the family of the child they kidnapped so they’ll take him back, because he’s such a monstrous little shit. Or, it can be the cold up-ending of the audience’s viewpoint, such as in The Sixth Sense, where (Do I really need to say SPOILER ALERT here?) the climax of the film makes clear that the protagonist, the unhappy child psychologist, is himself a ghost. It can even be simultaneously gruesome and blackly funny, as in Shakespeare’s Titus Andronicus, when (surprise, another SPOILER!) Titus reveals to his enemy Tamora both the recipe to his delicious meat pies and where her two sons went in the same sentence.
If you have your elements in place, a twist ending can be a satisfying way to usher your audience from the story with a bang, and give them something to talk about and remember for many a moon. Screw it up, and they might remember that too, but for entirely different (and unsatisfying) reasons. As with any potentially volatile element, use carefully and with all your ducks in a row.