My Favorite Mistake
One key distinction I always try to make when conducting in-house playtests is between the player’s consciously verbalized reactions and how they respond emotionally during play. Articulated responses can be invaluable, but require judicious sifting. In the well-known focus group effect, when people feel an obligation to make comments, or a desire to be heard, they will generate responses-which may or may not be in accord with their actual preferences. What you want from an experience and what you can articulate about can match up or diverge. To tell which is which, look at players’ nonverbal responses.
This isn’t always an easy thing to do. The observing designer must learn to look past his own emotional responses to accurately observe those of other participants. One’s own defensive urges can be as misleading as a player’s over-thought comments. Attachment to roleplaying theories and similar predigested intellectual frameworks can also serve as a barrier to understanding. This is as true for players as designers. Through forum discussion and the like, one might convince oneself that one mode of play is always objectively wrong, even if that is the style you might, in certain contexts, very much enjoy.
Divining state of mind becomes a special challenge when players are having an enjoyable, but internalized, experience. One factor I’ve underestimated more than once is the reward players derive from an extended character creation process. Intellectually I reckon that players want to get to the game as quickly as possible. But in certain games the creation process is at least as compelling as what comes afterwards. In fact, the game itself becomes richer when players have the time to fully engage their creativity.
As a GM, time spent during character creation can seem dull. You don’t get to join in until it’s over. That doesn’t mean the players aren't having a rich experience. The designer/GM must see past his own wandering attention to see how engrossed the players are. Prep can be a tedious slog, or it can be play. If it is play, a design might be ill-served by streamlining efforts that rush players through a process they’d sooner linger over.