The Rigors of Beta Testing: The Past, Present, and Future
There’s something magical about being accepted into a closed
beta test. Whether you’re a veteran of MMOs who finally
decided to jump into the testing portion of a game’s
development cycle or a relative virgin to the MMO experience that fell
in love with a well-known IP, receiving that
“You’re Invited!” email from a gaming
studio/publisher is like opening presents on Christmas Day.
You’re ready to work closely with the development team,
helping them to iron out bugs and improving game systems that
aren’t quite up to snuff. As the new kid in the testing
phase, you're prepared to formulate fantastic posts for the message
boards that argue about the functionality of certain mechanics, and you
expect thousands of bugs will be reported thanks to your efforts.
To answer these questions, Ten Ton Hammer extended an open invitation to the highest tier of MMO developers – those men and women that make the big decisions about beta tests – current and former high level employees of studios like Cryptic Studios, NetDevil, Sony Online Entertainment, Funcom, and Mythic Entertainment. While we may never know the inner workings behind some of past MMO development teams, these individuals can give us a glimpse into what their thought processes are when they start inviting the public into their testing ground.
To provide another viewpoint, the Ten Ton Hammer staff also took time to ask our legion of premium members about their thoughts on beta testing. As some of the most informed MMO fans on the planet, these individuals offer an interesting glimpse into the mind of the gamer. Their entries are as informative as they are entertaining. Once you're finished with the article, make sure you head on over to our forums and jot down your own thoughts on the beta testing process!
During the very beginning years of the graphical MMORPG movement, getting into early beta tests was as close as some of us may every get to being on a true development team. At this point in the late 1990s, the Internet was still the “latest new technological fad” and those gamers that had the tech savvy and intelligence to meander the often confusing spiderweb of download links, forum posts, and instruction manuals were greeted with games that were far from completion and teams that were eager to hear the opinions of their players. Countless stories were told on forums about how gamers were hired by companies simply because the individuals were intelligent and understood the systems behind these games.
For those players that were lucky (or unlucky) enough to get into the very first stages of the beta tests for games like Ultima Online, EverQuest or Asheron’s Call, these individuals found immense worlds that were almost devoid of players and provided the gamers with little direction on what to do after they created their character. It was truly up to the players to discover the ins-and-outs of the title, more often by trial-and-error than anything else.
But if you were active on the forums, played through the client often and tried to help the development team with in-game issues, you were often rewarded for your efforts with lengthy conversations, emails and forum responses from the developers. These initial testers were the first “game consultants” to the development teams of that era, and many of the most passionate gamers sought out (and found) jobs with MMO companies.
If you don’t believe me, just have a chat with Evan Michaels (System Designer for Age of Conan) or Sean Dahlberg (Community Manager for BioWare’s Star Wars: The Old Republic). Both of these individuals got their starts as forum junkies and fan site operators/participants for Anarchy Online and Shadowbane before moving on to find jobs in the industry. They were active members of their communities, and the development teams at Funcom and Wolfpack Studios took notice. In the current market, forum denizens are often lucky if they see any posts from their development team, and rarely get the chance to join forces with the studios they follow.
In fact, some modern gamers have even found that the beta forums for upcoming games provide them with nothing but stress and heartache. Forum trolls and flame artists aside, the development teams for many MMOs are often quick to squash any user that speaks out against their game or their beta testing procedures. A number of the Ten Ton Hammer premium members, many of whom are veteran MMO junkies, reported a variety of recent incidents that occurred during notable beta tests, where they were either harshly handled, had their “complaint” posts removed completely, or were simply insulted.
“While in [the Age of Conan beta forums] I spoke up, respectfully, about the class merges, about not testing sieges or PvP cities, and I wasn't just ignored or had a thread locked, I was flat out told by a developer on the forums to 'Shut up,'” Arkane reported. “I was shocked and I ended up returning my beta invite to them. The game launches and I find out that they still never tested those and when people finally got to do them on live it was a horror story.”
“While in the Warhammer beta, I had a developer flat out call me stupid and ignorant,” centrik responded. “There was a mix up, on their end mind you, and I received an email from them that caused some confusion. I brought it to their attention on the forums, to see if anyone else received this email. When no one did, I made a pretty harsh response. Mind you, the email was about being removed from the beta for NDA breech, so I was very, very concerned.”
These hardships are certainly nothing new for veteran gamers, but during the trials and tribulations of the "golden era," most gamers merely accepted the hardships that were presented to them as part of the testing process and did their best to help the development teams resolve their issues. By contrast, current gamers have become far less lenient with issues that show up in their testing experience. In an early interview during EverQuest’s final phase of beta testing, Brad McQuaid extolled the virtues of his beta community and how they surprised him:
In comparison, when asked similar questions during Vanguard’s later beta phase, McQuaid had this to say about his crop of beta testers:
“It’s okay,” he answered honestly. “I wish there were more people submitting bug reports, but that’s the way it goes with beta, and we’re still finding them regardless. Besides, I need all types [of players]. I need the exploits so we can find them...I need the jerks.”
The differences in the words, tone and attitude of the two statements is subtle, yet striking. In one interview, it was apparent that McQuaid was ecstatic with the response he had received from his testers; the other had him supporting the actions of “jerks” and issuing his “need” for these types of gamers to help him squash the bugs in his game. What occurred to the MMO player base between 1999 and 2006 that caused such a change of heart for this MMO developer?
After doing a bit of research, the focus of testing – and the focus of MMO players – had changed between the years the two games were released. While some gamers still existed that actually wanted to test his game in 2006, many players had grown more concerned with being able to take a “trial run” through unreleased MMOs. With those gamers in mind, the beta testing period of many other MMOs had turned into something more akin to an advertising campaign than a true testing period.
When asked a question about whether the MMO testing phases have changed in the past 5-10 years, the Ten Ton Hammer premium members suggested that beta tests are now almost solely about marketing and are populated with testers that ignore the “/bug” option and simply use the beta to demo the game. Even when the gamers do discuss the problems they see in the game, some MMO companies don't know how to handle the input these gamers provide.
“Honestly I feel that nowadays the developers get confused and can not find a happy medium between listening to every single testers opinion and listening to nobody,” Arkane suggested. “In some beta tests I have felt the community was highly ignored while in other tests I have felt that the developers tried to cater to every single tester. Both are and were recipes for failure.”
“Beta tests now are glorified demos for the games,” Condar answered. “It used to be so much different. I really hate what's been done to them, and the only way to be a ‘real’ tester now is to get into the alpha's or in some rare cases early closed betas.”
Besides the imminent dissolution of his company and inevitable heartbreak after the launch of his game, it’s safe to say that the environment surrounding MMOs changed drastically between the launch of Brad McQuaid's EverQuest and the release of Vanguard. With EverQuest, players hardly knew what to expect. Even those individuals that had played earlier graphical MMOs like Ultima Online and The Realm hadn’t jumped into a game like EverQuest before. It was a three dimensional experience based on late era MUDs and AD&D. Users hadn’t ever encountered each other in 3D spaces before, and thus the novelty of the game helped to overshadow the flaws and high barrier of entry into EQ. Gamers wanted that visual, visceral feeling of actually *seeing* a game of D&D come to life in front of them.
In an almost complete reversal of fortune, Vanguard launched into one of the most hostile markets ever seen in the MMO genre. Unlike the EverQuest beta, those individuals that were jumping into Vanguard were much more likely to have experienced any number of later day MMOs, including the massively popular World of Warcraft. While McQuaid constantly berated WoW, it was an inappropriate decision to simply ignore the success and advancements that WoW made to the genre. The gamers that arrived in the Vanguard beta needed an experience that at least had the look of potential surrounding it. From various accounts, the early Vanguard beta lacked so much content that trying to deduce any sort of “future” from the game was almost impossible.
But the hostility didn’t start (or end) with Vanguard. From the release of Anarchy Online to the Star Wars Galaxies CU-NGE, gamers have often been disappointed by the development teams that they want to succeed. Their anger and frustration began to circulate when it appeared that the development teams were ignoring their pleas for “more time” or “don’t change this” or “why?” Unfortunately, it’s often not the fault of the development teams making the MMOs, it’s the fact that MMOs require absurd amounts of cash to run and when outside publishers come into the picture, the gamers are often the individuals that feel the pain.
During the beta stage of MMOs, developers are under an inordinate amount of pressure. While the publishers shoulder the monetary strain, it’s truly up to the developers to actually produce a game that’s going to sell. If they don’t create something of worth, they can kiss their future contracts good-bye. Yet publishers rarely fork over bundles and bundles of cash to continue delaying a game until it’s “just right,” and so a hard release date may be internally set and the developer has to strive to hit that time period.
Other companies - Destination Games and the Earth and Beyond team for example - didn't survive poor beta tests or their leaks. One studio that did survive a poor beta test along with an eventual service cancellation and lived to tell the tale was NetDevil, a team that is now working on Jumpgate Evolution, LEGO Universe, and a few other unnamed projects. The studio's president, Scott Brown, previously talked with Ten Ton Hammer about beta tests and what developers face during that time period. Here’s what he had to say:
If there’s a beta that you go and play then you never play the game again, you know it’s probably not going to do so well. But if you play a beta and wish that the game was already launched because you don’t want to lose your character, you know that the game is going to be a hit.
It comes down to really simple stuff.
Unsurprisingly, NetDevil has opted to push back the Jumpgate Evolution beta period until the team believes that the game is *truly* ready for public consumption, because any preemptive release of the game could spell potential disaster. It’s a smart move, but one that many MMO companies can’t make due to struggling finances. Thus the feeling of helplessness and frustration continues on the part of the gamer. Despite their outcries, games continue to be released with numerous bugs, unfinished content, and poor game systems.
Perhaps one of the biggest examples of this happening occurred with the beta test and release of Age of Conan. After approximately five years of development time, it was rumored that the team at Funcom was running out of money and needed to push things out as quickly as possible.
“The problem was essentially too much to do in too little time,” our source said. “You had a team frantically trying to finish the game in the first place, then you had all these new and exciting bugs popping up. It wasn't that they didn't care. It was that there was too little time until launch and too many things that needed to be fixed.”
“I'd liken it to building a house,” the source continued, “and having your spouse wandering around telling you maybe this room should be blue and the couch should be over here, and that's all well and good but we needed to get the walls up first.”
When Ten Ton Hammer initially decided to craft this article, we sent out a list of questions to a number of prominent developer in the MMO industry. One of our most intriguing answers came from former EverQuest II senior producer and creative director, Scott Hartsman. His answer hit squarely on the money/beta testing issue, and rather than chopping his thoughts into pieces, here’s what he had to say about beta testing and money:
As product cost and complexity have increased, the emphasis of beta has indeed shifted toward toward marketing and load testing both your gameplay and operational systems. However, those are still critical activities in the high-budget, launch-big-or-die model. (That model has many weaknesses, but that's an entire topic in itself.)
The reason this happened is simple - It's about the money. Let's say you're a AAA game with 3-4 years of time and money invested, enough money to support a large team having worked on it for that long. Games like this frequently need to go for years before enough pieces come together before you can start making decisions about what's fun and what isn't.
By the time beta begins, you've made decision after decision that have compounded on each other. Your assumptions' assumptions' have assumptions about what your game is. The whole product, systems, content, operations, marketing, PR, community ramp, you name it -- is built upon them. Changing core assumptions about the product itself is unlikely to be possible without significant delays, costing progressively more money per month. (Remember, the months toward the end of the dev cycle are the most expensive ones by far.)
The game is, for the most part, what it is. You're capable of making shifts, but the more complex the game, the more minor the shifts you can make with any confidence. If assumptions that you made years ago turn out to be wrong, you're left to scramble, or in most cases, do your best to ameliorate the now-certain fallout.
If you haven't verified your gameplay at the point of having a beta, you've already left your fate to chance. (This is, of course, all presuming that your game has passed the technical bar in terms of stability, which is all too often not the case. And, again, is another flaw with the launch-big-or-die model.)
As budgets go up and schedules get longer, the model is growing more and more analogous to movies. If anything, people can see what goes on with blockbuster movie releases and draw certain comparisons.
No big beta? With a quality product at this stage in the industry's evolution the negatives almost never outweigh the positives.
Unlike movies, seldom are there a half dozen launches competing for attention in the same month, much less the same week, where movies might have some competitive advantage to keeping secrets this late in the game. MMOs differ from movies in that they're a long term time investment. The pattern of hype generation is different.
The way MMOs are most similar to movies, exploding costs aside, is that if you don't see an advance reviewer screening for a movie: Something Has Gone Terribly Wrong. Bad news is being kept out of the market in hopes of keeping day-one sales high.
The same can be said for lack of betas, repeatedly late betas, or overly-restrictive betas for MMOs.
The company knows that early sales are now where the bulk of the money is going to come from, instead of huge usage numbers over time, and it's doing what it needs to -- preserving those precious day one revenues, since it could well need that money to survive.
Things can still go terribly wrong for developers that choose to expose their games to the masses at a premature stage in the game’s development. Although Scott Hartsman argued that big betas are still necessary for the AAA titles with the multi-million dollar budgets, developers need to be wary of allowing the public to see their vision in a state that isn’t worthy of their time. Like a leak of a blockbuster movie or a pirated draft of an upcoming sequel to a bestselling book, the audience may ultimately decide that what they see in the early version of the product is what the final experience may be like.
In a later interview, Scott reiterated this thought again saying, “I no longer believe that beta’s purpose is to find bugs. I think beta’s purpose is to market your game. I mean it is the only way you can find things like huge balance issues and what happens when a bunch of players do something you never expected. It’s certainly there to find those things, but if you’re running beta for a game and you have crashes and bad frame rates, it’s not good.”
So while late game testing remains an important part of the development process, game companies need to insure that there games are in a “finished” or “near finished” state before they ever unleash their products to the public. And the proof is easily visible in two prominent (and cancelled) NCsoft titles: Tabula Rasa and Auto Assault.
As the numbers slowly revealed a lackluster launch for Richard Garriott’s Tabula Rasa, the legendary developer announced that inviting too many individuals into his game’s beta test caused a large amount of fallout and lost sales. Here’s what he told Gamasutra in a late 2007 article:
“We burned out some quantity of our beta-testers when the game wasn’t yet fun," he said, adding, "As we’ve begun to sell the game, the people who hadn’t participated in the beta became our fast early-adopters.”
Scott Brown echoed this sentiment in our late 2008 article with him:
Ten Ton Hammer: Really?
Nope. Never again. Never even came close. So many things about the game played so much better after that polish patch, but the numbers were never even close. That fact killed that game.
And these thoughts are continued by our own Ten Ton Hammer premium members. When asked about their worst beta experiences, most of them didn’t cite “unpolished content” as their major concerns. Instead, concerns with “being lied to” and “client not running” were top among the responders:
“Worst beta experience, for me at least, was with AoC,” Condar replied. “Seeing that Funcom had basically lied to us about what was in the game. Even those of us in the closed betas weren't given the correct information. That and trying to get the client to run some days was like having a second job.”
“Vanguard,” Thansal stated. “It didn't run.”
For those developers that have the money to do so, waiting until a game is ready to be viewed by the public eyes is essential. When asked about beta tests, Steve Perkins, the director of marketing for Mythic Entertainment and Warhammer Online, said this:
There is no science or rule book to follow for conducting a beta. It’s really a judgment call by the development team as to when to let players into their game. They need to decide if the game is ready for “primetime”. Make the wrong decision, and it could adversely affect your sales. Make the right decision, as DICE did with the Battlefield 1942 demo, and it could turn a new title into the “must have” game of the year!
While a good demo or beta, can really put a game on the map, not every title needs one. With the right media coverage and select consumer showcases (demoing at PAX, etc.), you can generate an incredible amount of positive buzz for your game. There have been many massive hits that never let the public touch the game before release.
A game beta is like a little morsel of food – if you find it delicious, you are going to want more and go out of your way to get more. If it leaves a bad taste in your mouth, you are not going back for seconds. Either way, you are likely to relate your experience to a friend and influence their decision as well.
So with the possibility of a beta test actually hurting the potential popularity of a game, is it even worth it for developers to stick their neck on the line and have beta tests? After hearing so much about the problems and issues developers face with beta, could they actually produce a game without any external testing? Is it even possible?
What does not scale are complex long term inter-relationships, but most games don't get bad beta results because of that. I think it's really a factor of being patient and waiting until you are ready before you test. There are all kinds of pressures that work against that and many of them are very real. I think the right time to test something is when you need the information from the test. If you do an internal test and it's not giving you useful data, you need to do external tests. When 300 people aren't enough to get useful data you need 1000, and so on. If you can manage to make an MMO that is awesomely fun without doing any testing, then I think that not showing the game won't hurt the success at all... that being said, I have no idea how you would make a good MMO without lots of testing: internal, external, and public.
According to Cryptic Studio's chief creative officer, Jack Emmert, there’s an enormous benefit to the developer that’s adaptable and can actually change things in their game during the beta phase. “In Champions Online, our beta has been great for identifying things that are not fun, and what the game is missing,” Emmert told Ten Ton Hammer. “We’ve even gone to our beta community for suggestions on how to solve certain issues. Now if the developer doesn’t have the time or the tools to make changes, then it’s impossible to adjust on the fly. Thankfully, Cryptic doesn’t fall into this category. I think closed beta has made a huge difference in our direction.”
But Jack also expanded on these questions in his next comment. "If a game is good, then beta or no, it’ll sell," he said. "If a team is a little unsure of how the public will react to something, then probably a beta can hurt. Someone might try the game and say, 'nah, not worth it.' That’s the real danger of beta; that potential customers will be able to decide before the purchase whether or not to buy. That’s why I think beta is vital to make changes to the game – to show the customers that you DO listen and that you want the game to succeed."
As Ten Ton Hammer continued to receive responses to our questions, it became readily apparent that almost all of the developers agree that some sort of beta is necessary. Whether its for marketing or stress testing or bug squashing, not having a beta isn't an option. They also agreed that if a game is good, and good when it hits a public beta, the game should sell no matter what. Here's what Jeff Hickman, executive producer of Warhammer Online, had to say about our beta questions:
Ten Ton Hammer also heard back from Turbine's Craig Alexander, VP of Product Development. His comments mainly focus on Lord of the Rings Online, but Turbine has had plenty of experience dealing with rougher beta tests and launches, such as those conducted for Asheron's Call 2 and Dungeons and Dragons Online. Compared to Mr. Hickman, the Turbine developer drew some more concise lines around how a developer can keep players from weighing too much of their final purchase decision on the results of their beta experience.
Alexander also wanted to point out that:
· How will new systems behave under load
· How well will new content handle “land rush” of players jumping to new content all at once
· How well will the design function with large populations
For this kind of Beta, the larger the group online at once the better.
Even after all that they've endured, it's obvious that the MMO community continues to want to beta test. But do they expect these games to be true beta tests or marketing campaigns? The responses we received from Ten Ton Hammer were very telling.
"Betas are obviously there to test the game for serious issues and bugs and to help eradicate them from the game, but over the years the hype for certain games has grown to substantial proportions because said games were in beta," Beerkeg said. "10 years ago there would have been no news about a game in beta until it had been released, and now we get new news every other week about said games that are still in beta."
"Beta tests have always been, without a doubt, about marketing as much as they have been about actual testing," centrik concluded. "It just is more evident today. However, from a tester point of view, I think there is a growing number of people who just want to try games before they come out."
Out of all the responses for that question from the Ten Ton Hammer members, our longtime member Annatar had the most passionate reply. His thoughts were clear and concise and obviously well thought out from his years of MMO experience.
"It is disappointing to say, but testing has become a bit of a joke, and I feel that the current crop of recent games are a reflection of that," he said. "So many games are being released incomplete (as far as hyped features go) and containing issues that should have been picked up and resolved during the closed beta phase at the latest, but this isn't happening."
"Perhaps developers should expect a bit more out of testers?" he continued. "I don't think the whole beta process would be hurt by developers outlining what is expected of testers to maintain their position with the testing 'team', and booting those that don't submit bug reports or feedback to make way for others who may be more inclined to take the process seriously, and not treat it as a free trial of the game."
"Market the game outside of the testing process, don't try and sell it to people via the testing phase when you need those testers to be making the product one worth selling," he finished."
There you have it folks. We started this article with the very beginnings of beta testing and concluded with your - the MMO gamer's - thoughts on the current crop of beta tests. So what conclusions can we draw from what we've learned?
Another important concept that's become clear is the fact that beta testing is different than it was in the past. Developers should not expect a large amount of gamers to help them design and develop major pieces of content in their game, especially if that content is in any way "broken" or "not fun." Early alpha and friends-and-family builds can certainly help set some of the content ideas, but by the time the developers get into a more public closed beta, they should have a fairly solid client that players are running on. Changes should be small and merely help to balance the final game and/or polish the content that already exists.
On the other side of the fence, gamers shouldn't expect their later stages of beta testing to include a lot of actual testing. While developers may have a focus for gamers to put their attention towards, the "big issues" just simply shouldn't exist in the later phases of beta. Triple-A MMOs will continue to move towards a heavy marketing focus in their betas, but development studios will hopefully learn from previous beta experiences and provide content that is fairly finished before it is unleashed to the public.
Thankfully, gamers will never have to worry about beta phases disappearing. Almost all of the developers stated that betas are essential to a game's development and final polishing, so this practice will continue into the future. But as we discussed earlier in the article, beta testing will continue to change over the next few years as teams learn from mistakes and really begin to implement the experiences that they learned from other teams mistakes. Everyone should keep their eyes on Champions Online and Jumpgate Evolution as these two games near their more "public" phases of beta. Will they learn from mistakes of the past?
Let's hope so.