Last year Ten Ton Hammer awarded
style="font-style: italic;">EVE Online
as the Best
Community in a MMOG of the year. Why? Because the community
knows what it wants and has had a fundamental part in designing the
future of the game. Ten Ton Hammer caught up with Arnar Gylfason,
Senior Producer of EVE
at gamescom 2010 to address some concerns that have been going about
the community. Lag, features backlog, and the future of the game are
our topics in this exclusive EVE
Online interview from gamescom 2010.
Ten Ton Hammer: The style="font-style: italic; font-weight: bold;">EVE style="font-weight: bold;"> community has brought up
concerns with the performance and
the lag with the most recent expansion. Can you comment on this?
Actually, this has come up a lot in recent interviews,
and naturally so. This is of concern to the community and a concern to
the developers as well.
Im not sure that youre familiar with the agile Scrum method of making
computer games. Basically, its teams of people working in two week
sprints. We have people who are extremely specialized in the networking
layer of the code and core mechanics of the code and would be the
perfect people to work on these problems. However, style="font-style: italic;">EVE is a complex
game and we have ten years of development code to try to figure out
where the bottlenecks are and what the problem is. Its not something
where you tell people: "you have two weeks; fix it!" Thats just not
going to happen.
alt="Amarr Fighter in EVE Online"
An Amarr Fighter in EVE
So we told these people, who were on scrum or other teams making
features or other stuff: "heres a list for you guys. You have to set
aside time, each with your own strengths, to prod, investigate, and
research what the code is doing and why."
I call this epiphany development. You're working on a specific problem.
You sort of know what it is. You have to get all sorts of data on it;
you have to get reproduction on the problem. Its all sort of iffy and
hazy, then one day, you have an epiphany on what the problem is and how
to fix it. Then actually fixing it takes a day of just coding, peer
reviewing the code, and testing. Then youre done. But the process
leading up to that can take days or weeks or even months of
investigating, painful reconstruction of scenarios, and it often
entails a great deal of community interaction.
Weve held mass tests where we get players on our test servers,
hundreds and hundreds of players, to recreate fleet battles to see why
this occurs in fleet battles. Why doesnt this happen to all of these
people fighting NPCs all at the same time? Whats going on here?
So, that was the set up that we had and it was working fairly well. We
were hammering out some fixes all of the time, but they werent
groundbreaking fixes to the lag. I shouldnt say 'fixes' because well
never fix lag; well just up the limit on what is acceptable, because
when we fix lag for 1000 people, then theyll just bring in 1200
people. If we fix that, then itll be 2000 people. So, incrementally
upping the limit on the number of people who can be in fleet battles is
something weve been doing for a while.
Now, something broke somewhere along the way, probably in the Dominion
expansion or shortly after the Dominion expansion, where the
performance just decreased all over. We had these people working on it
in this fashion since then, and theyve been investigating potential
causes. Theyve been fixing other problems. They find a lot of
side-line problems when theyre investigating these problems. So,
theyve been fixing that and publishing that.
Now, obviously, the problem has reached critical mass. The performance
is not up to snuff on what we consider a playable experience for our
fleet battles. The developers themselves are getting more frustrated
that they cant figure this out, or they feel sometimes that even if
they time set aside to do this, its not enough. So now, weve
formalized this team. Weve taken pretty much the most brilliant minds
and clustered the technology and network layering technology in
programming and put them together into one team and its sort of a
skunkworks team. They are allowed to do pretty much what they want and
try theories that they want. We will facilitate them getting it out and
getting it tested. The combined experience is probably forty, fifty
years in high-end clustering technology application programming.
We are hoping, and like I said, we cant give an exact time quote. I
cant say that yes, in two weeks, well fix lag because we have all of
these people working on it. But we do have all the people at CCP who
could be working on it are working on it dedicated 100 percent, and we
have teams of support personnel around them making sure that they have
everything they need, that they have the test servers they need, that
they have the reproduction stuff they need. We even have a separate
team building what we call thin clients, which is a client that runs on
a low CPU, low memory, and low ram, so that each computer can run tens
or hundreds of clients. We can recreate these fleet battles by
scripting these clients to behave as players would be running actual
clients. So, were able to basically make fleet fight in a can running
a small network of ten or twenty computers.
So, theres a lot of effort going into this, not just for the five or
six people who are actively working on identifying where the problem
lies and how to fix it, but there is a whole team of developers making
support tools for them. Then theres operational staff, customer
support staff, quality assurance staff, and community staff that are
supporting them. In all, we have about twenty-five people working on
Ton Hammer: You recently wrote a blog about community interaction and
community communication, and about this very thing.
I wrote a blog, as I do, about openly and transparently
telling the community how we split our resources between projects
Im the senior producer of EVE;
I dont know about the
other projects. All I know is that I borrow resources from them all the
time to work on my projects.
So I wrote this dev blog, and there I said that these teams are working
on features, these are working on Incarna, these are working on support
tools and internal tools, and these guys are working on fleet fights.
Probably, I wasnt clear enough that the people working on the fleet
fights are the same people. What has changed is their dedication and
commitment to that specific problem. At the same time, I think that
its important to understand that we could put every single developer
at CCP on this problem, but it wouldnt speed anything up. So, I think
it is better that we have a focused, dedicated group to do this.
Theyre the most brilliant minds we have. Adding more game designers to
the equation doesnt speed anything up.
Adding too many people is just like adding resistors to the pipeline.
It is my feeling that we are doing everything we can. We were doing a
lot, by taking these people out of their current teams, which actually
rattles production schedules for us, taking them out of their teams and
dedicating them 100 percent to this problem is pretty much as far as we
can go. Not because we want to go further, but because there isnt any
further to go. Now we have to just wait for that epiphany, and that
epiphany usually comes. Im not a software developer, but people in
software development tell me that the epiphany is like that shower
moment in the morning or that cup of coffee, where youve been thinking
about a problem for days and days and days, then you go home thinking
why cant I fix this? Then, you get in the shower or youre tucking
your kids into bed or whatever, you get it. It kind of clicks, and all
of a sudden, you go, of course! and its one line of code or changing
a variable somewhere. Thats what we have to do. We have to wait for
these epiphanies and see where they lead us.
Ten Ton Hammer: Can you
tell us about invasions? There was a recent developer post
that mentioned them and was then moderated.
Arnar Gylfason: Im
going a bit off the reservation here. I want the
team to say this themselves, but the Invasion feature is something that
would lend itself very well to other gameplay aspects of style="font-style: italic;">EVE.
Invasion, or Incursion, we havent decided upon a name, is encouraging
group gameplay through a mutual NPC enemy, but it can also attract
other player elements. You can get a group of people together that
youve never met in a social environment to attack a sort of rampaging
horde of NPCs, and that is one part of the gameplay.
Another part of the gameplay is that the pirates will win and kill you
all, and this is to the heart of EVE,
which is to not mandate or
dictate the content on the players, but rather provide interesting
scenarios and see where they take it, and usually they take it to the
extreme. Thats something were very happy with. The mechanic of this
is something that will lend itself well to factional warfare, where you
have a problem of not obtaining critical mass for a fleet fight in the
right locations, so we can use similar mechanics for this. Im not a
programmer, but it feels like copy-paste programming to me. To apply
this to factional warfare or other parts of style="font-style: italic;">EVE, so this is
we would be iterating in the future, and then with twists and turns,
turning it from being a new feature to being an expansion to older
Ton Hammer: What about the backlog of features? The community has been
for a elements that haven't yet been added. Where does this backlog fit
in on the prioritization list?
Currently, Ive been updating my list on people who are
working on EVE
development a week ago, and its 140 people. Thats game
designers, QA, software, content creation, and art. So, we have 140
people working on it. Theres, obviously, a strong specialization
fields for each of them, but you still reach bottlenecks in the
production when you have a few guys who understand database
programming, or you have a few designers who understand social design.
So even with a throughput of 140 people, it doesnt mean that we can do
everything and have it go through as fast. So we have to sort of pick
Some of the features that have been requested, or need some loving,
theoretically, we have enough people to work on alongside everything
else, but there are bottlenecks in the production chain where things
fall apart. What we are doing, however, is were taking our focus off
these features, in general, to refactoring older features. This is a
process that started, basically, at the beginning of this year and will
continue throughout next year till the point where well halt
production on new features, and turn to refactoring the old features or
updating older game designs.
You can call it development eras if you want. So you add a bunch of new
features, then you reiterate on those features until youre satisfied
that theyre polished. In general, what you see is, and I guess the
players have been seeing this, and they will be seeing it more in the
next release and the next release thereafter is a focus shift to
Pirates in EVE
Ten Ton Hammer: So that
ties into the final question. Whats next?
Well, weve been working on an the Incarna system for
years. Four years, to be exact (stop laughing). We feel that weve
reached a critical mass in development there, and thats in full
production. Thats something to be looking for very shortly. I wouldnt
want to commit to it for this expansion and I wont commit to it for
this expansion, but moving into next year, absolutely, youll be seeing
some amazing things on the Incarna front.
Then youll have the Incursion feature, or the Invasion feature. Were
iterating on the planetary interaction feature as well that we launched
earlier this year, and we have plans on iterating on the combat and
tactical UIs at some point next year.
Thats the plan, but plans can sometimes change. Plans will change, but
currently our plan is that we want to iterate and update the tactical
UI for fleet fights. You can always fix live for this particular set of
problems, but having to render hundreds or thousands of ships onscreen,
or we can take it down a notch and say, when youre in a fleet fight
this big, perhaps you need a different view. You just need to render it
different. So thats something we want to move into and have a certain
level of detail loading of combat views based upon the magnitude of the
battle youre in. Thats a sort of game design change that can help lag
a lot in the future, but its a fairly big refactoring of "Excel in
Space." I dont think players want to lose their "Excel in Space."
To read the latest guides, news, and features you can visit our EVE Online Game Page.