Sideways, not Up.
While I'm working on my current game, I've been discussing an MMO with one of my friends, and how I find the idea of 'nerfing' content as new content is added as feeling a bit wrong. As part of the conversation, I suggested that the characters in the game should reach a plateau, and make it less about progressing in power, and more about exploration and refining what you already have.
This has me thinking about the RPG I'm working on, and RPG design in general. Rather than having characters just gain more and more powerful, why not work on a game which refines what the character already has? In this case, it would mean attribute gain (if even possible) is limited - much like how in Shadowrun 1st edition, characters could only increase each attribute through karma once.
But the focus would be on skills and capabilities. A character may get a new skill, but that would mean simply that the skill has been gained -- and not much more. The next step would be refinement. The character can take a specialty in the skill, representing a more focussed and detailed awareness. Perhaps the character can take more specialties, but each one costs more. General knowledge is easy, but specialized knowledge is harder, and becoming specialized in a number of fields takes time and effort.
A magic system might work on the same principles. You learn general knowledge in a magical field, and then you refine what you know. Evocation is good. Fire magic is better. Fireball is best. As you refine the process, the 'cost' to use the magic is lowered, or you're generally more effective. Specializing in more fields of magic costs more over time, and you're a better magician for it. Not 'omg powerful', but you're better at what you know.
A character's power is based on what they know and how well they know how to use it. A character who spreads out his knowledge may be more versatile, but lacking oomph, while a character who concentrates and specializes is better within her fields, but may not be as flexible.
I think the trick is that after a certain point, things slow down significantly. As the character develops, the costs go up universally. You gained evocation. Necromancy is going to cost more. Divination is going to cost more than that. Illusion will cost yet more. Specializing in what you have already is a bit cheaper, but each additional specialty costs more than the last. The idea is -- a character is only able to learn and develop so much.
This makes gear important. You have your limits, but tools will help more. A 'magic sword' is useful, because it allows you to surpass your limitations. A magic sword may give you the skill to use it automatically, or augment your skill if you already have it. It is a universally useful effect, which never fades over time -- it saved you having to put your (very) precious XP into something, and you can put that someplace else.
This gives the added bonus of making threats remain fairly consistent. Rather than having the characters face ever-increasing spirals of danger, the same creatures are dangerous at the start of the game, and only a little less dangerous later on. If you fight a lot of goblins, you're probably dumping your XP into skills that help you survive your fight with goblins. You probably also get gear to make you more effective at fighting goblins. But you can only get so good, and your gear can only get you so far. Enough goblins, and you'll get mowed down.
I'm thinking I might work on this in greater detail.