There’s something magical about being accepted into a closed
beta test. Whether you’re a veteran of MMOs who finally
decided to jump into the testing portion of a game’s
development cycle or a relative virgin to the MMO experience that fell
in love with a well-known IP, receiving that
“You’re Invited!” email from a gaming
studio/publisher is like opening presents on Christmas Day.
You’re ready to work closely with the development team,
helping them to iron out bugs and improving game systems that
aren’t quite up to snuff. As the new kid in the testing
phase, you're prepared to formulate fantastic posts for the message
boards that argue about the functionality of certain mechanics, and you
expect thousands of bugs will be reported thanks to your efforts.

style="margin: 10px; border-collapse: collapse; float: right; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/40317"> src="/image/view/40317/preview"
style="border: 2px solid ; width: 200px;" />

But in this modern era of MMO gaming, do your efforts really make a
difference on the final design of the game? Is it possible to convince
a developer that the game truly isn’t ready to launch? That
they should polish the title for another 2-3 months to work out the
kinks? Or are those items set in stone as soon as a game reaches any
sort of “public” beta testing stage?

To answer these questions, Ten Ton Hammer extended an open invitation
to the highest tier of MMO developers – those men and women
that make the big decisions about beta tests – current and
former high level employees of studios like Cryptic Studios, NetDevil,
Sony Online Entertainment, Funcom, and Mythic Entertainment. While we
may never know the inner workings behind some of past MMO development
teams, these individuals can give us a glimpse into what their thought
processes are when they start inviting the public into their testing
ground.

To provide another viewpoint, the Ten Ton Hammer staff also took time
to ask our legion of premium members about their thoughts on beta
testing. As some of the most informed MMO fans on the planet, these
individuals offer an interesting glimpse into the mind of the gamer.
Their entries are as informative as they are entertaining. Once you're
finished with the article, make sure you head on over to our forums and
jot down your own thoughts on the beta testing process!

During the very beginning years of the
graphical MMORPG movement,
getting into early beta tests was as close as some of us may every get
to being on a true development team. At this point in the late 1990s,
the Internet was still the “latest new technological
fad” and those gamers that had the tech savvy and
intelligence to meander the often confusing spiderweb of download
links, forum posts, and instruction manuals were greeted with games
that were far from completion and teams that were eager to hear the
opinions of their players. Countless stories were told on forums about
how gamers were hired by companies simply because the individuals were
intelligent and understood the systems behind these games.

style="margin: 10px; border-collapse: collapse; float: left; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/65353"> src="/image/view/65353/preview"
style="border: 2px solid ; width: 200px;" />

However, beta testing in that era was often a challenging experience.
These were unfinished games with incomplete rules and unpolished
content. Hardware upgrades were often necessary to simply run many of
these titles, especially games like the 3D-accelerator necessary
href="http://www.tentonhammer.com/taxonomy/term/38"
target="_blank">EverQuest. Game installation
seldom went smoothly, and many players
found themselves wondering if they would ever find a way to get into
beta. Once gamers were playing in the new world, they’d often
fall through the landscape, zone into a “nothing”
space, or simply experience other strange issues due to the relatively
new technology powering these titles.

For those players that were lucky (or unlucky) enough to get into the
very first stages of the beta tests for games like href="http://www.tentonhammer.com/taxonomy/term/154"
target="_blank">Ultima
Online
,
EverQuest
or target="_blank">Asheron’s
Call
, these individuals found immense
worlds that were almost devoid of players and provided the gamers with
little direction on what to do after they created their character. It
was truly up to the players to discover the ins-and-outs of the title,
more often by trial-and-error than anything else.

But if you were active on the forums, played through the client often
and tried to help the development team with in-game issues, you were
often rewarded for your efforts with lengthy conversations, emails and
forum responses from the developers. These initial testers were the
first “game consultants” to the development teams
of that era, and many of the most passionate gamers sought out (and
found) jobs with MMO companies.

If you don’t believe me, just have a chat with Evan Michaels
(System Designer for target="_blank">Age
of Conan
) or Sean Dahlberg (Community Manager
for BioWare’s href="http://www.tentonhammer.com/taxonomy/term/1422"
target="_blank">Star Wars: The Old Republic). Both
of these
individuals got their starts as forum junkies and fan site
operators/participants for href="http://www.tentonhammer.com/taxonomy/term/40"
target="_blank">Anarchy
Online
and Shadowbane
before moving
on to find jobs in the industry. They were active members of their
communities, and the development teams at Funcom and Wolfpack Studios
took notice. In the current market, forum denizens are often lucky if
they see any posts from their development team, and rarely get the
chance to join forces with the studios they follow.

In fact, some modern gamers have even
found that the beta forums for
upcoming games provide them with nothing but stress and heartache.
Forum trolls and flame artists aside, the development teams for many
MMOs are often quick to squash any user that speaks out against their
game or their beta testing procedures. A number of the Ten Ton Hammer
premium members, many of whom are veteran MMO junkies, reported a
variety of recent incidents that occurred during notable beta tests,
where they were either harshly handled, had their
“complaint” posts removed completely, or were
simply insulted.

style="margin: 10px; border-collapse: collapse; float: right; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/42956"> src="/image/view/42956/preview"
style="border: 2px solid ; width: 200px;" />

target="_blank">Warhammer
was frustrating because you could not voice
anything in the closed forums, anything bad would be deleted
immediately,” RawGutts answered.

“While in [the Age
of Conan
beta forums] I spoke up,
respectfully, about the class merges, about not testing sieges or PvP
cities, and I wasn't just ignored or had a thread locked, I was flat
out told by a developer on the forums to 'Shut up,'” Arkane
reported. “I was shocked and I ended up returning my beta
invite to them. The game launches and I find out that they still never
tested those and when people finally got to do them on live it was a
horror story.”

“While in the Warhammer
beta, I had a developer flat out call
me stupid and ignorant,” centrik responded. “There
was a mix up, on their end mind you, and I received an email from them
that caused some confusion. I brought it to their attention on the
forums, to see if anyone else received this email. When no one did, I
made a pretty harsh response. Mind you, the email was about being
removed from the beta for NDA breech, so I was very, very
concerned.”
 
These hardships are certainly nothing new for veteran gamers, but
during the trials and tribulations of the "golden era," most gamers
merely accepted the hardships that were presented to them as part of
the testing process and did their best to help the development teams
resolve their issues. By contrast, current gamers have become far less
lenient with issues that show up in their testing experience. In an
early interview during EverQuest’s
final phase of beta
testing, Brad McQuaid
target="_blank">extolled the virtues of his beta
community and how they surprised him:

style="font-style: italic;">The game continues to amaze me
virtually every day. The teamwork I see
developing, and the tactics used by long time testers in some of our
higher level dungeons is truly amazing. The dynamic we've created
between the various classes, each with their own strengths and
weaknesses, turned out to be even greater than I'd imagined. Another
example of something that blew me away have been some of the in-game
marriages that have taken place, where over 60 people have shown up to
watch the ceremony. Watching a community form up from nothing into
something in a virtual world you've helped create is truly an awesome
experience.

In comparison, target="_blank">when asked similar questions
during target="_blank">Vanguard’s
later beta phase, McQuaid had this to say about his crop of beta
testers:

“It’s okay,” he answered honestly.
“I wish there were more people submitting bug reports, but
that’s the way it goes with beta, and we’re still
finding them regardless. Besides, I need all types [of players]. I need
the exploits so we can find them...I need the jerks.”

The differences in the words, tone and attitude of the two statements
is subtle, yet striking. In one interview, it was apparent that McQuaid
was ecstatic with the response he had received from his testers; the
other had him supporting the actions of “jerks” and
issuing his “need” for these types of gamers to
help him squash the bugs in his game. What occurred to the MMO player
base between 1999 and 2006 that caused such a change of heart for this
MMO developer?

After doing a bit of research, the focus
of testing – and the focus of MMO
players – had changed between the years the two games were
released. While some
gamers still existed that actually wanted to test his game in 2006,
many
players had grown more concerned with being able to take a
“trial run” through unreleased MMOs. With those
gamers in
mind, the beta testing period of many other MMOs had turned into
something more akin to an advertising campaign than a true testing
period. 

When asked a question about whether the MMO testing
phases have changed in the past 5-10 years, the Ten Ton Hammer premium
members suggested that beta tests are now almost solely about marketing
and are populated with testers that ignore the
“/bug” option and simply use the beta to demo the
game. Even when the gamers do discuss the problems they see in the
game, some MMO companies don't know how to handle the input these
gamers provide.

style="margin: 10px; border-collapse: collapse; float: left; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/62919"> src="/image/view/62919/preview"
style="border: 2px solid ; width: 200px;" />

“[Beta] seems to be more about marketing the game and hyping
it up,” RawGutts said.

“Honestly I feel that nowadays the developers get confused
and can not find a happy medium between listening to every single
testers opinion and listening to nobody,” Arkane suggested.
“In some beta tests I have felt the community was highly
ignored while in other tests I have felt that the developers tried to
cater to every single tester. Both are and were recipes for
failure.”

“Beta tests now are glorified demos for the games,”
Condar answered. “It used to be so much different. I really
hate what's been done to them, and the only way to be a
‘real’ tester now is to get into the alpha's or in
some rare cases early closed betas.”

Besides the imminent dissolution of his company and inevitable
heartbreak after the launch of his game, it’s safe to say
that the environment surrounding MMOs changed drastically between the
launch of Brad McQuaid's EverQuest
and the release of Vanguard.
With
EverQuest,
players hardly knew what to expect. Even those individuals
that had played earlier graphical MMOs like style="font-style: italic;">Ultima Online and style="font-style: italic;">The Realm
hadn’t jumped into a game like style="font-style: italic;">EverQuest before.
It was a
three dimensional experience based on late era MUDs and AD&D.
Users hadn’t ever encountered each other in 3D spaces before,
and thus the novelty of the game helped to overshadow the flaws and
high barrier of entry into EQ. Gamers wanted that visual, visceral
feeling of actually *seeing* a game of D&D come to life in
front of them.

In an almost complete reversal of fortune, style="font-style: italic;">Vanguard launched
into one
of the most hostile markets ever seen in the MMO genre. Unlike the
EverQuest
beta, those individuals that were jumping into style="font-style: italic;">Vanguard were
much more likely to have experienced any number of later day MMOs,
including the massively popular href="http://wow.tentonhammer.com/" target="_blank"> style="font-style: italic;">World of Warcraft.
While McQuaid
constantly berated WoW, it was an inappropriate decision to simply
ignore the success and advancements that WoW made to the genre. The
gamers that arrived in the Vanguard
beta needed an experience that at
least had the look of potential surrounding it. From various accounts,
the early Vanguard
beta lacked so much content that trying to deduce
any sort of “future” from the game was almost
impossible.

But the hostility didn’t start (or end) with style="font-style: italic;">Vanguard. From
the release of Anarchy
Online
to the href="http://www.tentonhammer.com/taxonomy/term/42"
target="_blank">Star
Wars Galaxies
CU-NGE, gamers have
often been disappointed by the development teams that they want to
succeed. Their anger and frustration began to circulate when it
appeared that the development teams were ignoring their pleas for
“more time” or “don’t change
this” or “why?” Unfortunately,
it’s often not the fault of the development teams making the
MMOs, it’s the fact that MMOs require absurd amounts of cash
to run and when outside publishers come into the picture, the gamers
are often the individuals that feel the pain.

During the beta stage of MMOs, developers
are under an inordinate
amount of pressure. While the publishers shoulder the monetary strain,
it’s truly up to the developers to actually produce a game
that’s going to sell. If they don’t create
something of worth, they can kiss their future contracts good-bye. Yet
publishers rarely fork over bundles and bundles of cash to continue
delaying a game until it’s “just right,”
and so a hard release date may be internally set and the developer has
to strive to hit that time period.

style="margin: 10px; border-collapse: collapse; float: right; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/64398"> src="/image/view/64398/preview"
style="border: 2px solid ; width: 200px;" />

So, more often than not, the biggest amount of pressure for a
development company comes during the beta testing period. Even during
the closed beta experience, word of a game that isn’t fun or
is absurdly broken will spread. For gamers, the threat of NDAs almost
doesn’t exist. There were dozens of videos on YouTube
showcasing Warhammer
Online
and Age
of Conan
before the release of
those two games. While we may never know the true impact the beta leaks
from those two games had on their end sales figures, it's a safe bet
that at least some money was lost thanks to negative feedback from beta
leaks that occurred during the testing.

Other companies - Destination Games and the Earth and Beyond team for
example - didn't survive poor beta tests or their leaks. One studio
that did
survive a poor beta test along with an eventual service
cancellation and lived to tell the tale was NetDevil, a team that is
now working on href="http://www.tentonhammer.com/taxonomy/term/306"
target="_blank">Jumpgate
Evolution
, href="http://www.tentonhammer.com/taxonomy/term/170"
target="_blank">LEGO
Universe
, and a few other
unnamed projects. The studio's president, Scott Brown, href="http://www.tentonhammer.com/node/51644" target="_blank">previously
talked with Ten Ton Hammer
about beta tests and what developers face during that time period.
Here’s what he had to say:

style="font-style: italic;">Basically, the pressure is
almost all in beta. I mean,
there’s some pressure at launch, but most gamers know if a
game is going to be successful way before the actual launch. Right? You
just know.



If there’s a
beta that you go and play then you never play
the game again, you know it’s probably not going to do so
well. But if you play a beta and wish that the game was already
launched because you don’t want to lose your character, you
know that the game is going to be a hit.

style="font-style: italic;" />


It comes down to really
simple stuff.

Unsurprisingly, NetDevil has opted to push back the Jumpgate Evolution
beta period until the team believes that the game is *truly* ready for
public consumption, because any preemptive release of the game could
spell potential disaster. It’s a smart move, but one that
many MMO companies can’t make due to struggling finances.
Thus the feeling of helplessness and frustration continues on the part
of the gamer. Despite their outcries, games continue to be released
with numerous bugs, unfinished content, and poor game systems.

Perhaps one of the biggest examples of
this happening occurred with the
beta test and release of Age
of Conan
. After approximately five years
of development time, it was rumored that the team at Funcom was running
out of money and needed to push things out as quickly as possible.

style="margin: 10px; border-collapse: collapse; float: left; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/41704"> src="/image/view/41704/preview"
style="border: 2px solid ; width: 200px;" />

Under the safety of anonymity, one ex-Funcom employee recently talked
to Ten Ton Hammer about the days leading up to the final stages of AoC
development. It’s a sad story, but one that probably happens
more often than we think.

“The problem was essentially too much to do in too little
time,” our source said. “You had a team frantically
trying to finish the game in the first place, then you had all these
new and exciting bugs popping up. It wasn't that they didn't care. It
was that there was too little time until launch and too many things
that needed to be fixed.”

“I'd liken it to building a house,” the source
continued,
“and having your spouse wandering around telling you maybe
this room should be blue and the couch should be over here, and that's
all well and good but we needed to get the walls up first.”

When Ten Ton Hammer initially decided to
craft this article, we sent
out a list of questions to a number of prominent developer in the MMO
industry. One of our most intriguing answers came from former href="http://eq2.tentonhammer.com/" target="_blank"> style="font-style: italic;">EverQuest II
senior producer and creative director, Scott Hartsman. His answer hit
squarely on the money/beta testing issue, and rather than chopping his
thoughts into pieces, here’s what he had to say about beta
testing and money:

style="margin: 10px; border-collapse: collapse; float: right; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/65166"> src="/image/view/65166/preview"
style="border: 2px solid ; width: 200px;" />

For the AAA, eight- and
nine- figure budget extravaganzas, big betas
aren't going away any time soon.  What companies get out of
them has shifted over time, but they remain an important part of
getting a game out the door.

style="font-style: italic;" />


As product cost and
complexity have increased, the emphasis of beta has
indeed shifted toward toward marketing and load testing both your
gameplay and operational systems.  However, those are still
critical activities in the high-budget, launch-big-or-die
model.   (That model has many weaknesses, but that's
an entire topic in itself.)

style="font-style: italic;" />


The reason this happened
is simple - It's about the money. 
Let's say you're a AAA game with 3-4 years of time and money invested,
enough money to support a large team having worked on it for that
long.  Games like this frequently need to go for years before
enough pieces come together before you can start making decisions about
what's fun and what isn't.

style="font-style: italic;" />


By the time beta begins,
you've made decision after decision that have
compounded on each other.  Your assumptions' assumptions' have
assumptions about what your game is.  The whole product,
systems, content, operations, marketing, PR, community ramp, you name
it -- is built upon them.  Changing core assumptions about the
product itself is unlikely to be possible without significant delays,
costing progressively more money per month.  (Remember, the
months toward the end of the dev cycle are the most expensive ones by
far.)




The game is, for the
most part, what it is.  You're capable of
making shifts, but the more complex the game, the more minor the shifts
you can make with any confidence.  If assumptions that you
made years ago turn out to be wrong, you're left to scramble, or in
most cases, do your best to ameliorate the now-certain fallout.

style="font-style: italic;" />


If you haven't verified
your gameplay at the point of having a beta,
you've already left your fate to chance.  (This is, of course,
all presuming that your game has passed the technical bar in terms of
stability, which is all too often not the case.  And, again,
is another flaw with the launch-big-or-die model.)

style="font-style: italic;" />


As budgets go up and
schedules get longer, the model is growing more
and more analogous to movies.  If anything, people can see
what goes on with blockbuster movie releases and draw certain
comparisons.




No big beta? 
With a quality product at this stage in the
industry's evolution the negatives almost never outweigh the positives.




Unlike movies, seldom
are there a half dozen launches competing for
attention in the same month, much less the same week, where movies
might have some competitive advantage to keeping secrets this late in
the game.  MMOs differ from movies in that they're a long term
time investment.  The pattern of hype generation is different.

style="font-style: italic;" />


The way MMOs are most
similar to movies, exploding costs aside, is that
if you don't see an advance reviewer screening for a movie: 
Something Has Gone Terribly Wrong.  Bad news is being kept out
of the market in hopes of keeping day-one sales high.

style="font-style: italic;" />


The same can be said for
lack of betas, repeatedly late betas, or
overly-restrictive betas for MMOs.

style="font-style: italic;" />


The company knows that
early sales are now where the bulk of the money
is going to come from, instead of huge usage numbers over time, and
it's doing what it needs to -- preserving those precious day one
revenues, since it could well need that money to survive.

Things can still go terribly wrong for
developers that choose to expose
their
games to the masses at a premature stage in the game’s
development. Although Scott Hartsman argued that big betas are still
necessary for the AAA titles with the multi-million dollar budgets,
developers need to be wary of allowing the public to see their vision
in a state that isn’t worthy of their time. Like a leak of a
blockbuster movie or a pirated draft of an upcoming sequel to a
bestselling book, the audience may ultimately decide that what they see
in the early version of the product is what the final experience may be
like.

style="margin: 10px; border-collapse: collapse; float: left; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/64036"> src="/image/view/64036/preview"
style="border: 2px solid ; width: 200px;" />

If we turn an eye back to NetDevil and style="font-style: italic;"
href="http://www.tentonhammer.com/taxonomy/term/31"
target="_blank">Auto Assault, at their
target="_blank">post-mortem talk on the game at OGDC 2007,
Hermann Peterscheck and Scott
Brown both stated that beta tests should simply be used for marketing
and stress testing. A development team shouldn’t rely on beta
testing to point out all the flaws in a game’s design,
because that simply won’t happen. Technical problems can be
ascertained through.

target="_blank">In a later interview, Scott
reiterated this thought again saying, “I no longer
believe that beta’s purpose is to find bugs. I think
beta’s purpose is to market your game. I mean it is the only
way you can find things like huge balance issues and what happens when
a bunch of players do something you never expected. It’s
certainly there to find those things, but if you’re running
beta for a game and you have crashes and bad frame rates,
it’s not good.”

So while late game testing remains an important part of the development
process, game companies need to insure that there games are in a
“finished” or “near finished”
state before they ever unleash their products to the public. And the
proof is easily visible in two prominent (and cancelled) NCsoft titles:
style="font-style: italic;">Tabula Rasa
and Auto Assault.

As the numbers slowly revealed a
lackluster launch for Richard
Garriott’s Tabula Rasa
, href="http://www.gamasutra.com/php-bin/news_index.php?story=16490"
target="_blank">the legendary developer announced
that
inviting too many individuals into his game’s beta test
caused a large amount of fallout and lost sales. Here’s what
he told Gamasutra in a late 2007 article:

style="margin: 10px; border-collapse: collapse; float: right; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/14183"> src="/image/view/14183/preview"
style="border: 2px solid ; width: 200px;" />

“I actually think the biggest mistake was made not by the
marketing department, but by the development team. We invited too many
people into the beta when the game was still too broken.”

“We burned out some quantity of our beta-testers when the
game wasn’t yet fun," he said, adding, "As we’ve
begun to sell the game, the people who hadn’t participated in
the beta became our fast early-adopters.”

Scott Brown target="_blank">echoed this sentiment in our late
2008 article with him:

style="font-style: italic;">We have a bunch of meters that
we use to monitor buzz and interest in
the game, and when we went out with our first Auto Assault beta and
realized that we had a ton more work to do, we shut the beta down for
awhile. We did a great big polish patch, and everything played way
better, but when we reopened the game again we never had the same
number of people visiting the web page or downloading the client ever
again.



Ten
Ton Hammer: Really?




Nope. Never again. Never
even came close. So many things about the game
played so much better after that polish patch, but the numbers were
never even close. That fact killed that game.

style="font-style: italic;">But it wasn’t closing
the beta that caused the problem, it
was starting the beta too soon. We fooled ourselves into thinking it
was ready to go.

And these thoughts are continued by our own Ten Ton Hammer
premium members. When asked about their worst beta experiences, most of
them didn’t cite “unpolished content” as
their major concerns. Instead, concerns with “being lied
to” and “client not running” were top
among the responders:

“Worst beta experience, for me at least, was with
AoC,” Condar replied. “Seeing that Funcom had
basically lied to us about what was in the game. Even those of us in
the closed betas weren't given the correct information. That and trying
to get the client to run some days was like having a second
job.”

“Vanguard,” Thansal stated. “It didn't
run.”

For those developers that have the money
to do so, waiting until a game
is ready to be viewed by the public eyes is essential. When asked about
beta tests, Steve Perkins, the director of marketing for Mythic
Entertainment and Warhammer
Online
, said this:

style="margin: 10px; border-collapse: collapse; float: left; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/61550"> src="/image/view/61550/preview"
style="border: 2px solid ; width: 200px;" />

style="font-style: italic;">Ultimately, you need to believe
in your product before you conduct any
sort of open beta or release a demo. You can message to players all you
want that the game is a “work in progress” and that
many things will change before final release, but that won’t
stop them from making judgments about the title based on their beta
experiences.



There is no science or
rule book to follow for conducting a beta.
It’s really a judgment call by the development team as to
when to let players into their game. They need to decide if the game is
ready for “primetime”. Make the wrong decision, and
it could adversely affect your sales. Make the right decision, as DICE
did with the Battlefield 1942 demo, and it could turn a new title into
the “must have” game of the year!

style="font-style: italic;" />


While a good demo or
beta, can really put a game on the map, not every
title needs one. With the right media coverage and select consumer
showcases (demoing at PAX, etc.), you can generate an incredible amount
of positive buzz for your game. There have been many massive hits that
never let the public touch the game before release.

style="font-style: italic;" />


A game beta is like a
little morsel of food – if you find it
delicious, you are going to want more and go out of your way to get
more. If it leaves a bad taste in your mouth, you are not going back
for seconds. Either way, you are likely to relate your experience to a
friend and influence their decision as well.

So with the possibility of a beta test
actually hurting the potential
popularity of a game, is it even worth it for developers to stick their
neck on the line and have beta tests? After hearing so much about the
problems and issues developers face with beta, could they actually
produce a game without any external testing? Is it even possible?

style="margin: 10px; border-collapse: collapse; float: right; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/33575"> src="/image/view/33575/preview"
style="border: 2px solid ; width: 200px;" />

After we posed this question to our select group of developers, Hermann
Peterscheck, producer for the "soon to be in beta" style="font-style: italic;">Jumpgate Evolution,
was one of the first individuals to chime in. If you've read any of our
previous interviews with Hermann, you know that his thoughts are clear
and well-written, so his response to our inquiries wasn't a surprise at
all.

style="font-style: italic;">Well. It's a harsh world.
Whenever you put your baby out there, you
risk that people say it's ugly.  I can't speak specifically to
Tabula Rasa, but we've certainly experienced unfavorable feedback
during testing periods. I think that it's impossible to know exactly
how your game will be received but by doing increasingly large play
tests you can get a pretty good idea of what people think. I'm amazed
at how well the tests scale. If 10/40 people like or don't like
something, generally 100/400 are about the same.
style="font-style: italic;" />


What does not scale are
complex long term inter-relationships, but most
games don't get bad beta results because of that. I think it's really a
factor of being patient and waiting until you are ready before you
test. There are all kinds of pressures that work against that and many
of them are very real. I think the right time to test something is when
you need the information from the test. If you do an internal test and
it's not giving you useful data, you need to do external tests. When
300 people aren't enough to get useful data you need 1000, and so on.
If you can manage to make an MMO that is awesomely fun without doing
any testing, then I think that not showing the game won't hurt the
success at all... that being said, I have no idea how you would make a
good MMO without lots of testing: internal, external, and public.

According to Cryptic Studio's chief creative officer, Jack Emmert,
there’s an enormous benefit to the developer that’s
adaptable and can actually change things in their game during the beta
phase. “In href="http://www.tentonhammer.com/taxonomy/term/860"
target="_blank">Champions
Online
, our beta has been great for
identifying things that are not fun,  and what the game is
missing,” Emmert told Ten Ton Hammer.
“We’ve even gone to our beta community for
suggestions on how to solve certain issues. Now if the developer
doesn’t have the time or the tools to make changes, then
it’s impossible to adjust on the fly. Thankfully, Cryptic
doesn’t fall into this category. I think closed beta has made
a huge difference in our direction.”

But Jack also expanded on these questions in his next comment. "If a
game is good, then beta or no, it’ll sell," he said. "If a
team is a little unsure of how the public will react to something, then
probably a beta can hurt. Someone might try the game and say, 'nah, not
worth it.' That’s the real danger of beta; that potential
customers will be able to decide before the purchase whether or not to
buy. That’s why I think beta is vital to make changes to the
game – to show the customers that you DO listen and that you
want the game to succeed."

As Ten Ton Hammer continued to receive
responses to our questions, it
became readily apparent that almost all of the developers agree that
some sort of beta is necessary. Whether its for marketing or stress
testing or bug squashing, not having a beta isn't an option. They also
agreed that if a game is good, and good when it hits a public beta, the
game should sell no matter what. Here's what Jeff Hickman, executive
producer of Warhammer Online, had to say about our beta questions:

style="margin: 10px; border-collapse: collapse; float: left; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/66862"> src="/image/view/66862/preview"
style="border: 2px solid ; width: 200px;" />

style="font-style: italic;">I truly believe that beta should
NEVER hurt the sales of any good
game.  Beta not only helps the developers test the game, but
also gives the players insight into how great the game is and helps
build buzz amongst the community.  Unless you are concerned
that players won’t actually *like* your game, then a well run
beta is a must in my mind. *winks* Letting as many players as possible
give feedback on your game and also getting as many as possible to talk
within the community about your game is very important.

Ten Ton Hammer also heard back from Turbine's Craig Alexander, VP of
Product Development. His comments mainly focus on href="http://lotro.tentonhammer.com/" target="_blank"> style="font-style: italic;">Lord of the Rings Online,
but Turbine has
had plenty of experience dealing with rougher beta tests and launches,
such as
those conducted for Asheron's
Call 2
and target="_blank">Dungeons
and Dragons Online
. Compared to Mr. Hickman, the
Turbine developer drew some more concise lines around how a developer
can keep players from weighing too much of their final purchase
decision on the results of their beta experience.

style="font-style: italic;">The only time a Beta can hurt a
game is if the game is not ready when it enters Beta.  It all
comes down to managing player expectations in terms of what they can
expect in Beta, and how it does/does not relate to the game they will
buy at launch.  Clear, constant communication is the key, as
well as demonstrating a facility to understand feedback, and respond
effectively and appropriately.  As usual, it all comes down to
giving the game the time is needs to become a great game and truly
ready for launch.  As long as players see that commitment from
you, they will be flexible in their assessments during Beta.

Alexander also wanted to point out that:

style="font-style: italic;">There is significant value to
Beta for a number of reasons.  The scope of Beta depends on
the scope of the release.  In each case, Beta is valuable for
load testing and stress testing of content as well as new
systems.  Turbine’s platform is battle-tested and
very stable overall, so our load testing is really focused on issues
such as:



· How will
new systems behave under load

style="font-style: italic;" />


· How well
will new content handle “land rush” of players
jumping to new content all at once

style="font-style: italic;" />


· How well
will the design function with large populations

style="font-style: italic;" />


For this kind of Beta,
the larger the group online at once the better.

For
more significant launches like our original launch of Shadows of Angmar
or Mines of Moria, we have extended Beta tests to learn about how
players will perceive the new systems and content to see how we can
make the game better before official launch.  This was
instrumental in the quality of what we launched with Shadows of Angmar
and Moria.  Many valuable lessons are learned during Beta that
we have created process and technology to capture and then leverage
into our game. 

Even after all that they've endured, it's obvious that the MMO
community continues to want to beta
test. But do they expect these games to be true beta tests or marketing
campaigns? The responses we received from Ten Ton Hammer were very
telling.

"Betas are obviously there to test the game for serious issues and bugs
and to help eradicate them from the game, but over the years the hype
for certain games has grown to substantial proportions because said
games were in beta," Beerkeg said. "10 years ago there would have been
no news about a game in beta until it had been released, and now we get
new news every other week about said games that are still in beta."

"Beta tests have always been, without a doubt, about marketing as much
as they have been about actual testing," centrik concluded. "It just is
more evident today. However, from a tester point of view, I think there
is a growing number of people who just want to try games before they
come out."

Out of all the responses for that question from the Ten Ton Hammer
members, our longtime member Annatar had the most passionate reply. His
thoughts were clear and concise and obviously well thought out from his
years of MMO experience.

"It is disappointing to say, but testing has become a bit of a joke,
and I feel that the current crop of recent games are a reflection of
that," he said. "So many games are being released incomplete (as far as
hyped features go) and containing issues that should have been picked
up and resolved during the closed beta phase at the latest, but this
isn't happening."

"Perhaps developers should expect a bit more out of testers?" he
continued. "I don't think the whole beta process would be hurt by
developers outlining what is expected of testers to maintain their
position with the testing 'team', and booting those that don't submit
bug reports or feedback to make way for others who may be more inclined
to take the process seriously, and not treat it as a free trial of the
game."

"Market the game outside of the testing process, don't try and sell it
to people via the testing phase when you need those testers to be
making the product one worth selling," he finished."

There you have it folks. We started this
article with the very
beginnings of beta testing and concluded with your - the MMO gamer's -
thoughts on the current crop of beta tests. So what conclusions can we
draw from what we've learned?

style="margin: 10px; border-collapse: collapse; float: right; width: 200px;"
border="1">

href="http://www.tentonhammer.com/node/64983"> src="/image/view/64983/preview"
style="border: 2px solid ; width: 200px;" />

First and foremost, beta testers and community members want to have at
least a reasonable amount of honest communication with the development
team for the game they are testing. However, it seems like gamers are
reasonable about their expectations. They want ways to focus their
attention, and they want to be able to outline the problems that they
see within the game either through forum posts or feedback reports.
Probably most important of all, they want to see progress on the game
that they're playing that is coming as a result of their testing.

Another important concept that's become clear is the fact that beta
testing is different than it was in the past. Developers should not
expect a large amount of gamers to help them design and develop major
pieces of content in their game, especially if that content is in any
way "broken" or "not fun." Early alpha and friends-and-family builds
can certainly help set some of the content ideas, but by the time the
developers get into a more public closed beta, they should have a
fairly solid client that players are running on. Changes should be
small and merely help to balance the final game and/or polish the
content that already exists.

On the other side of the fence, gamers shouldn't expect their later
stages of beta testing to include a lot of actual testing. While
developers may have a focus for gamers to put their attention towards,
the "big issues" just simply shouldn't exist in the later phases of
beta. Triple-A MMOs will continue to move towards a heavy marketing
focus in their betas, but development studios will hopefully learn from
previous beta experiences and provide content that is fairly finished
before it is unleashed to the public.

Thankfully, gamers will never have to worry about beta phases
disappearing. Almost all of the developers stated that betas are
essential to a game's development and final polishing, so this practice
will continue into the future. But as we discussed earlier in the
article, beta testing will continue to change over the next few years
as teams learn from mistakes and really begin to implement the
experiences that they learned from other teams mistakes. Everyone
should keep their eyes on href="http://www.champions-online.com/user/register"
target="_blank">Champions
Online
and Jumpgate
Evolution
as
these two games near their more "public" phases of beta. Will they
learn from mistakes of the past?

Let's hope so.


To read the latest guides, news, and features you can visit our Vanguard: Saga of Heroes Game Page.

Last Updated: Mar 29, 2016

Comments