It started when I was thinking about George Lucas talking about why he killed off Obi-Wan Kenobi, I guess. Originally Obi-Wan was going to survive the original Star Wars, but as time went on, George Lucas realized that he didn't have anything to do as a character after they escaped the Death Star, and further, it made the bad guys look impotent for the good guys to escape so unscathed. So anyway, I was thinking that in order for drama to really reach its potential, I think there has to be a price the good guys pay. If the good guys win with no loss, it rings false and makes for a flat story. Simple wish-fulfillment fantasies, even well-written ones like Edgar Rice Burroughs' works, don't do this and have a more visceral form of drama. I feel wish fulfillment fantasy kind of peaks low, in the area of drama.
Anyway, I was looking at the books on my shelf and my DVDs and novels, and I was surprised at how often the most drama correlated to a sense of loss or defeat. I think it's that low-high contrast that creates the drama. Tolkien was a master of putting his characters through hell, then raising them back up triumphantly. It doesn't have to be a clean loss; to use Tolkien again, look at the Witch-King killing Theoden.
As the gates of Gondor are battered down and all hope seems lost, the cavalry from Rohan arrives and drives back the bad guys. We already have a low-high note. In this moment of triumph, however, a dart hits Theoden's horse, which rolls over him, crushing him. As the Witch-King, lord of the Nazgul, moves to finish him, Eowyn bars his way -- an enemy even the reborn Gandalf fears to face. (Pretty bold to fly into an army that's just defeated yours, intending to kill their king with impunity, isn't it?) Eowyn defeats him, but at a great cost: her and Merry both have injured arms and are taken out of the fight, and Theoden dies shortly afterwards. The fact that they had to sacrifice so much sells you on the idea that the Witch-King is a powerful foe, even though you don't really see him do much. Even though this portion of the book is little more than two pages long, it's one of the most dramatic things I've ever read.
The loss doesn't have to be lasting, or even real. In the above example, Merry and Eowyn recovered. A character can seem to die, but come back later, if done properly. (It's done improperly very often in modern comic books, which has no dramatic impact.) It does have more impact if it counts, though. But it must seem like the protagonists, who we've hopefully come to sympathize with, have lost something significant and dear. Let's look at some of those examples from my shelves:
* Harry Potter is good about this. Look at the end of The Goblet of Fire. After making it to the end of the Tri-Wizard Tournament, he decides to make it a tie with his rival, the likeable, honorable Cedric Diggory. This leads immediately to Cedric's death, as they are transported to the rebirth of Voldemort. Moreover, though Harry narrowly escapes with his life, he learns that the thing that protected him from Voldemort is gone; his safety net is gone and now he must somehow survive the wrath of perhaps the most powerful wizard ever on his natural talents, in spite of his youth. The Order of the Pheonix has a similar loss, with the death of Sirius Black. They stop the villians from attaining their goal, but at great loss. In a series, a story can end on a down note, and recover before the end.
* The Lion, the Witch, and the Wardrobe uses an allegory for Jesus in this way. (Is the story of Jesus an example of my postulation? Maybe!) Aslan sacrifices his life willingly, suffering much humiliation and abuse, but his sacrifice pays off. This is an example of a fake sacrifice, as Aslan comes back. It still works, though.
* The Princess Bride is FULL of these moments. ("What do you mean, 'Westley dies'?") Westley also comes back, though. If you've only watched the movie, the book doesn't let them off too easily, either. It ends thusly, with them being pursued but at an advantage: "However, this was before Inigo's wound reopened; and Westley relapsed again; and Fezzik took the wrong turn; and Buttercup's horse threw a shoe. And the night behind them was filled with the cresendoing sound of pursuit..." Nobody gets a free meal in The Princess Bride, even when they're unreasonably awesome, like Westley.
* Le Morte d'Arthur has the Knights of the Round Table going off to find the Holy Grail, which leads to the end of the golden age of Camelot, with many of the knights dying and pretty much all suffering greatly. The promised paragon knight, Galahad, basically retires from being a knight upon attaining the grail. And when they come back, evil has trickled into Camelot, leading to many more deaths, Guineveire having to flee and become a nun, and ultimately the death of King Arthur at the hands of his treacherous bastard son, Modred. There is no upbeat to this, but it is a tragic romance (Romance: 1a. A medieval tale based on legend, chivalrous love and adventure, and the supernatural), so it doesn't have to have an upbeat ending. Tragedies are a lost art form. They were very popular in ancient Greece, for example.
* Moby-Dick is a pretty straight forward adventure story until they sight the white whale. It ends, of course, with the entire crew being killed except Ishmael. It is also a tragedy, being a cautionary tale against hubris, I think.
* Sherlock Holmes dies (albeit temporarily) in The Final Problem, slain by his nemesis, Professor Moriarty. This is actually an example of the principle I've outlined being done badly! We never hear anything about Professor Moriarty until this story, at which point Sherlock Holmes builds him up as an evil version of himself: his ultimate nemesis. However, we never see Professor Moriarty do anything to earn this reputation and he dies at the end, so he ends up as a simple tool to remove Sherlock Holmes, since Arthur Conan Doyle was tired of writing about him. Arthur Conan Doyle wrote the next Sherlock Holmes story, The Hound of the Baskervilles, a long while afterwards, but it was set before Sherlock Holmes died. When he finally did revive Sherlock Holmes, he came up with a very contrived explanation, further undermining the integrity of The Final Problem.
* Heck, let's even look at my comic books. Two of the most famous super hero stories, the Dark Phoenix Saga and the death of Gwen Stacy, revolve around the deaths of major characters. Frank Miller's run on Daredevil was full of loss. Since comic books are serialized, they can really take their time with this. Chris Claremont, like Tolkien, really knew how to lay his characters low, so that they could rise up again. Storm loses her powers, and her link to nature, and it is several years before her powers return. And when they do return, it is a real moment of triumph, because of the build-up: we missed her powers, with that feeling being enhanced by her suffering. (That's the sympathizing with the protagonists thing I mentioned earlier.)
* Season 2 of the Avatar: the Last Airbender has the heroes soundly defeated and sent on the run. Aang is mortally wounded, and his avatar power is apparently sealed forever. The Earth Kingdom has fallen after many decades of holding out against the Fire Nation. Zuko, who was so close to turning himself around and becoming a good person, has taken a big step back into villainy. This is the most dramatic point in the show.
* The point where The Maxx cartoon ends is a surprising turn. After establishing this story setup, the female lead, Julie, throws in the towel and leaves town. The story can't work without her. This is one of the most dramatic turns in a storyline I've encountered. It really torpedoes the series as we know it. It's like if Princess Peach said to hell with Mario and Bowser and fled to parts unknown. Mario would have to just be a plumber, and Bowser would have to take up chess. But they made it work! The only problem is that you need to read the comics to see what happens next.
* Seasons 1 and 2 of The Venture Bros. end with high drama. Season 1 ends with the title characters dying, for crying out loud! (Man, that was one long wait for season 2!) Season 3 ends with everything changing. Brock quits being the family's bodyguard. The clones are all destroyed, so the brothers have no fallback in case of death. The henchman who sounds like Ray Romano dies. Seasons 2 and 4 had good finales, but they lacked the drama of the aforementioned ones, and hey, they lacked a sense of loss. I don't think it's a coincidence.
* Season 2 of X-Men Evolution ends with the team's identities being revealed to the public and half the team being captured. The former changes the face of the show.
* Jeez, I picked up Bambi this year and it might have one of the most famous losses of all: Bambi's mom. It doesn't get much more dramatic than that.
* Blade Runner has Deckard Kain losing his humanity, literally and figuratively, as the movie progresses. He quickly realizes how unsavory his job is, and (in newer cuts, especially) it seems likely that he is a replicant himself. He can live with the latter, but not the former.
* Jason and the Argonauts has the crew being slain left and right. Hercules, the most powerful of the Argonauts, departs from the group almost right away.
* Metropolis has the revolt of the lower class, causing great destruction and mayhem. Only in the face of this can an accord be reached. (The storm before the calm!)
* In The Nightmare Before Christmas, Jack hits a low as all the effort he put forth and all his dreams are shattered. But then he picks himself up and sets everything right. When he's first inspired to take over Christmas, he immediately casts aside Christmas balls, shattering them and bursts some Christmas lights on his electric chair. This is some nice symbolic foreshadowing, as he takes something pretty and destroys it. Zero even seems alarmed.
* Pan's Labyrinth has a dramatic point where everything seems to be going fine, then Ofelia ruins it with a moment of weakness: eating at the monster's table, when she was explicitly told not to. She fails the faun's test, who tells her she will not see him again. At the same time, her mother seems doomed to die, leaving her in the hands of her evil step father. Indeed, to ultimately succeed, she must sacrifice her life. (This is a fascinating movie, as the three plot threads -- Ofelia's real world troubles, her fantastic adventures, and the other woman's story (forgot her name) all reflect each other.)
* The Empire Strikes Back shows this principle very well indeed. The entire movie is about loss, leading to triumph in Return of the Jedi being much sweeter. The rebels lose their base and are forced to flee, right at the start, setting the tone of the film. At the end, they've lost Han, and Luke's lost his hand and his innocence. They've gained very little in exchange. Return of the Jedi has them triumphing. Some of the folks involved wanted to kill off a character, Han or Lando, but George Lucas was adamant that they not kill any of the heroes. However, Darth Vader's death is a loss, in that he was redeemed, but paid for it in with his life, so it wasn't as clean a victory as some may think. It still was lacking the punch of some of the other films, though, I feel.
* Fullmetal Alchemist's core theme is pretty much this principle! The brothers lose one of their bodies and the other's arm and leg, sacrificed in an attempt to undo another loss without payment. They tried to create an anti-climax to the tragedy to their mother's death, but instead became tragic figures themselves. In Fullmetal Alchemist, nothing's free, and something must be sacrificed for something to be gained. The climax uses this principle a few times to extremely dramatic effect, but I won't spoil it for those who haven't seen it.
* Princess Mononoke has the forces of nature battling the enroachment of civilization. Neither side will accept anything except the destruction of the other side. Only the cataclysmic destruction of the forest spirit, leading to the destruction of both the forest and the fort, can pursuade them to live in harmony.