The following was published in the Nov 2011 issue of Game Developer magazine…

“You have to design for success, plan for failure, and then know how to rebound from that failure. Our 1.0 version of AI War was successful but not wildly so. Little irritations added up and annoyed people enough that they stopped playing. When the public gets the game, they find problems that you never did, and you must devote time to fixing them. Once you do, the fans are happy, and the game becomes more successful than it would have been otherwise. You have to eat a lot of humble pie, but after awhile you get really used to that, and the stuff that used to give me a hard time emotionally when I was first starting out is just par for the course now. I don’t even notice. It’s just feedback, and whether it’s viable or not determines if it goes in the ‘yes’ bucket or the ‘no’ bucket or the ‘maybe’ bucket.”

– Chris Park, designer of AI War: Fleet Command, from the Three Moves Ahead podcast, Episode 37

To be a game designer is to be wrong. Ideas do not work out as planned. Certain mechanics prove tedious instead of fun. Players spend their time focusing on the “wrong” parts of the game. The original vision starts to slowly slip away as it come into contact with the gamer.

Furthermore, designers are often bombarded with suggestions – from the rest of the team, from vocal fans, from well-meaning friends, and from dreaded executive fly-bys. Faced with such a deluge, the natural instinct is often to defend the design. These suggestions are simply trials and tribulations to be overcome so that you, the designer, can continue making the game “your” way.

The problem is that processing feedback is a fundamental part of the game design process, as important as the original vision itself. Games are not static objects that can be observed or judged in a vacuum. Instead, they live in the minds of our players, which means that each one might experience the game in a different way.

Thus, designers need to be great listeners more than great persuaders. If a designer ever finds herself needing to explain why a player should be having fun when he is not, something has gone seriously wrong. Instead, designers must listen to players with a great sense of humility, with an understanding that this feedback is the only way to remove the fog which separates the game in the designer’s head from the one that is actually playable.

Ultimately, games must speak for themselves, so designers need to learn not to rely on the crutch of their own enthusiasm and communications skills to sell their ideas. Dropping one’s ego to learn from the criticism can be an emotional challenge for designers who identify too closely with their original vision. Instead, they need to place faith in the design process itself, not in their inevitably doomed first draft.

Listen to the Team?

Thus, gathering and assessing feedback is one of the crucial skills for a modern game designer. The first, and sometimes only, source of feedback is the team itself. Any committed development team should be ready and eager to play its own game and provide the designer crucial early feedback on what works and what doesn’t. However, feedback from the team carries significant limitations.

To begin, the team lives with the game for months, probably even years, far beyond an average player’s time with the game. As veterans of development process, they can play on auto-pilot, making themselves blind to unintuitive mechanics or confusing UI. Developers quickly lose the ability to see the game objectively, often believing that the game is either in better or in worse shape than it really is.

Further, team members are often hired for their special skills – 3D animation, sound design, network optimization – and not because of their passion for the product itself. They might forget some of the simple joys that new players will still experience when starting the game. More dangerously, they might burnout on the game altogether and begin to resent the intense demands of your most vocal community members.

Trust the Community?

The community, of course, can be an overflowing cauldron of ideas and suggestions. When developing Civilization 3, our most active fansite presented us with “The List” – an exhausting, 200,000 word tome detailing their expectations for the upcoming game. Wading into the forums can be an overwhelming experience for most designers, requiring a thick skin as posters rip apart their development choices.

However, no one understands a game better than players who are dedicated enough to join a community and make the game a part of their social life. These gamers might play for hundreds of hours, gaining knowledge of mechanics and systems that elude even the designers themselves.

The challenge with forums is that what players say and what they actually do are often two different things. In a talk at GDC 2011, Ben Cousins described just such a situation with the free-to-play online shooter Battlefield Heroes. The game had not been generating enough revenue, so the team reworked the monetization system – making it harder for non-paying players to “rent” premium items and then beginning to sell these items directly for cash.

The change caused an uproar in the forums; within a week, a 4,000-post thread developed decrying the changes, with many veterans pledging to quit the game. Exacerbating the debate was the fact that Cousins had publicly declared years earlier that “we have no plans to sell weapons.” The press picked up the controversy, leading to articles such as “Battlefield Heroes is Practically Ruined” on Kotaku.

The metrics, however, told a very different story; revenue tripled with no discernible decrease in active users. It is hard to tell if the posters who pledged to quit were actually lying or not, but they were clearly not representative of the average player. Cousins dug deeper and found that only 20% of players had ever visited the forums and that only 2% had ever actually posted a message.

Furthermore, compared with the silent majority, community members had a much higher conversion rate (27% vs. 2%) and ARPPU ($110 vs. $32). Thus, the thoughts expressed on the forums were an inaccurate and misleading representation of the player base’s actual feelings. The posters were perhaps making threats to change the game as they wanted (to save themselves money) instead of revealing their actual beliefs.

The Heroes experience highlights the importance of metrics as a secondary source of feedback. Watching what players actually do can be as important as listening to what they have to say. Still, metrics have their limitations as no set of numbers is going to help the designer understand why people have stopped playing the game.

While measuring how often unit X is built instead of unit Y provides a valuable tool for balancing an RTS, it’s not necessarily clear that making the two units equally viable will actually make the game more fun. Metrics are great at answering specific objective questions that require real data – what difficulty level is most commonly picked first? – but to learn whether a game is actually fun, the designer’s only option is to find out what players are feeling by listening closely to what they are saying.

Find Your Voices

Thus, designers are left with the conundrum that their best source of feedback – the vocal fan community – is not only an unreliable source of information, but one that might be actively trying to mislead the developers. Perhaps most frustrating is the possibility that the more forum posters are aware that the team is listening, the more likely they are to lie to the designers to get what they want. MMO developers are familiar with the player type who will always argue that his or her character’s class is woefully underpowered, all objective evidence to the contrary.

This problem can be handled with a more proactive approach to gathering fan feedback. Not all fans are the same – a precious few are able to see the forest and provide accurate feedback that speaks to the health of the game’s overall experience. While developing Civilization 4, we cultivated just such a group of enlightened fans to provide feedback we could trust.

These players had a history of being reliable sources of information during the post-release development of Civ 3, and we provided them with a special, private forum for direct communication with the team. This group became our primary source of feedback both before and after release, providing us with much greater certainty about which ideas were working and which ones were not; Civ 4 would have been significantly differently – and certainly worse – without their input.

These groups must be managed carefully, however, to prevent the members from developing a sense of entitlement or superiority over other players. For this reason, the group’s existence should be, if possible, a closely guarded secret. Further, the developers must try their best to find a representative group of players, perhaps looking outside the forums for new members.

Listen Early, Listen Often

Another accurate way to gather feedback is with “Kleenex” testing, so named because players get access to the pre-release game once and are then thrown away. The valuable lessons here come from players’ initial reactions to the game, before they become accustomed to UI holes or gameplay quirks. Valve famously runs these tests regularly by gathering up random players from local game stores.

However, depth testing is also important, which can only be achieved by giving players continual access to the game before release, to explore and experiment with the game’s systems and mechanics. Big publishers often have trouble giving fans early access to their games, for fear of leaking cracked versions to pirate sites or spreading confidential information to rival publishers.

Indies developers actually have a big advantage here because their greatest danger is not security, but obscurity. Thus, many recent indies (Spelunky, Desktop Dungeons, The Wager) have released early versions of their games, generating both marketing buzz and valuable feedback.

Some indie games, such as Frozen Synapse, Minecraft, and Spy Party, have even generated revenue by selling access to these alphas. This option gives teams the chance to bootstrap their way along while also learning how the game performs in the wild, a great option to help fight the long odds that most indies face.