Last year Ten Ton Hammer awarded href="http://www.tentonhammer.com/eve"> style="font-style: italic;">EVE Online
as the Best
Community in a MMOG of the year. Why? Because the community
knows what it wants and has had a fundamental part in designing the
future of the game. Ten Ton Hammer caught up with Arnar Gylfason,
Senior Producer of EVE
at gamescom 2010 to address some concerns that have been going about
the community. Lag, features backlog, and the future of the game are
our topics in this exclusive EVE
Online interview from gamescom 2010.

---

Ten Ton Hammer: The style="font-style: italic; font-weight: bold;">EVE style="font-weight: bold;"> community has brought up
concerns with the performance and
the lag with the most recent expansion. Can you comment on this?
style="font-weight: bold;">


Arnar Gylfason:
Actually, this has come up a lot in recent interviews,
and naturally so. This is of concern to the community and a concern to
the developers as well.



I’m not sure that you’re familiar with the agile Scrum method of making
computer games. Basically, it’s teams of people working in two week
sprints. We have people who are extremely specialized in the networking
layer of the code and core mechanics of the code and would be the
perfect people to work on these problems. However, style="font-style: italic;">EVE is a complex
game and we have ten years of development code to try to figure out
where the bottlenecks are and what the problem is. It’s not something
where you tell people: "you have two weeks; fix it!" That’s just not
going to happen.

So we told these people, who were on scrum or other teams making
features or other stuff: "here’s a list for you guys. You have to set
aside time, each with your own strengths, to prod, investigate, and
research what the code is doing and why."



I call this epiphany development. You're working on a specific problem.
You sort of know what it is. You have to get all sorts of data on it;
you have to get reproduction on the problem. It’s all sort of iffy and
hazy, then one day, you have an epiphany on what the problem is and how
to fix it. Then actually fixing it takes a day of just coding, peer
reviewing the code, and testing. Then you’re done. But the process
leading up to that can take days or weeks or even months of
investigating, painful reconstruction of scenarios, and it often
entails a great deal of community interaction.



We’ve held mass tests where we get players on our test servers,
hundreds and hundreds of players, to recreate fleet battles to see why
this occurs in fleet battles. Why doesn’t this happen to all of these
people fighting NPCs all at the same time? What’s going on here?



So, that was the set up that we had and it was working fairly well. We
were hammering out some fixes all of the time, but they weren’t
groundbreaking fixes to the lag. I shouldn’t say 'fixes' because we’ll
never fix lag; we’ll just up the limit on what is acceptable, because
when we fix lag for 1000 people, then they’ll just bring in 1200
people. If we fix that, then it’ll be 2000 people. So, incrementally
upping the limit on the number of people who can be in fleet battles is
something we’ve been doing for a while.



Now, something broke somewhere along the way, probably in the Dominion
expansion or shortly after the Dominion expansion, where the
performance just decreased all over. We had these people working on it
in this fashion since then, and they’ve been investigating potential
causes. They’ve been fixing other problems. They find a lot of
side-line problems when they’re investigating these problems. So,
they’ve been fixing that and publishing that.



Now, obviously, the problem has reached critical mass. The performance
is not up to snuff on what we consider a playable experience for our
fleet battles. The developers themselves are getting more frustrated
that they can’t figure this out, or they feel sometimes that even if
they time set aside to do this, it’s not enough. So now, we’ve
formalized this team. We’ve taken pretty much the most brilliant minds
and clustered the technology and network layering technology in
programming and put them together into one team and it’s sort of a
skunkworks team. They are allowed to do pretty much what they want and
try theories that they want. We will facilitate them getting it out and
getting it tested. The combined experience is probably forty, fifty
years in high-end clustering technology application programming.



We are hoping, and like I said, we can’t give an exact time quote. I
can’t say that yes, in two weeks, we’ll fix lag because we have all of
these people working on it. But we do have all the people at CCP who
could be working on it are working on it dedicated 100 percent, and we
have teams of support personnel around them making sure that they have
everything they need, that they have the test servers they need, that
they have the reproduction stuff they need. We even have a separate
team building what we call thin clients, which is a client that runs on
a low CPU, low memory, and low ram, so that each computer can run tens
or hundreds of clients. We can recreate these fleet battles by
scripting these clients to behave as players would be running actual
clients. So, we’re able to basically make fleet fight in a can running
a small network of ten or twenty computers.



So, there’s a lot of effort going into this, not just for the five or
six people who are actively working on identifying where the problem
lies and how to fix it, but there is a whole team of developers making
support tools for them. Then there’s operational staff, customer
support staff, quality assurance staff, and community staff that are
supporting them. In all, we have about twenty-five people working on
this now.

Ten
Ton Hammer: You recently wrote a blog about community interaction and
community communication, and about this very thing.
style="font-weight: bold;">


Arnar Gylfason:
I wrote a blog, as I do, about openly and transparently
telling the community how we split our resources between projects
within EVE.
I’m the senior producer of EVE;
I don’t know about the
other projects. All I know is that I borrow resources from them all the
time to work on my projects.



So I wrote this dev blog, and there I said that these teams are working
on features, these are working on Incarna, these are working on support
tools and internal tools, and these guys are working on fleet fights.
Probably, I wasn’t clear enough that the people working on the fleet
fights are the same people. What has changed is their dedication and
commitment to that specific problem. At the same time, I think that
it’s important to understand that we could put every single developer
at CCP on this problem, but it wouldn’t speed anything up. So, I think
it is better that we have a focused, dedicated group to do this.
They’re the most brilliant minds we have. Adding more game designers to
the equation doesn’t speed anything up.

Adding too many people is just like adding resistors to the pipeline.
It is my feeling that we are doing everything we can. We were doing a
lot, by taking these people out of their current teams, which actually
rattles production schedules for us, taking them out of their teams and
dedicating them 100 percent to this problem is pretty much as far as we
can go. Not because we want to go further, but because there isn’t any
further to go. Now we have to just wait for that epiphany, and that
epiphany usually comes. I’m not a software developer, but people in
software development tell me that the epiphany is like that shower
moment in the morning or that cup of coffee, where you’ve been thinking
about a problem for days and days and days, then you go home thinking
why can’t I fix this? Then, you get in the shower or you’re tucking
your kids into bed or whatever, you get it. It kind of clicks, and all
of a sudden, you go, “of course!” and it’s one line of code or changing
a variable somewhere. That’s what we have to do. We have to wait for
these epiphanies and see where they lead us.



Ten Ton Hammer: Can you
tell us about invasions? There was a recent developer post
that mentioned them and was then moderated.
style="font-weight: bold;">


Arnar Gylfason: I’m
going a bit off the reservation here. I want the
team to say this themselves, but the Invasion feature is something that
would lend itself very well to other gameplay aspects of style="font-style: italic;">EVE.



Invasion, or Incursion, we haven’t decided upon a name, is encouraging
group gameplay through a mutual NPC enemy, but it can also attract
other player elements. You can get a group of people together that
you’ve never met in a social environment to attack a sort of rampaging
horde of NPCs, and that is one part of the gameplay.



Another part of the gameplay is that the pirates will win and kill you
all, and this is to the heart of EVE,
which is to not mandate or
dictate the content on the players, but rather provide interesting
scenarios and see where they take it, and usually they take it to the
extreme. That’s something we’re very happy with. The mechanic of this
is something that will lend itself well to factional warfare, where you
have a problem of not obtaining critical mass for a fleet fight in the
right locations, so we can use similar mechanics for this. I’m not a
programmer, but it feels like copy-paste programming to me. To apply
this to factional warfare or other parts of style="font-style: italic;">EVE, so this is
something
we would be iterating in the future, and then with twists and turns,
turning it from being a new feature to being an expansion to older
features.

Ten
Ton Hammer: What about the backlog of features? The community has been
waiting
for a elements that haven't yet been added. Where does this backlog fit
in on the prioritization list?
style="font-weight: bold;">


Arnar Gylfason:
Currently, I’ve been updating my list on people who are
working on EVE
development a week ago, and it’s 140 people. That’s game
designers, QA, software, content creation, and art. So, we have 140
people working on it. There’s, obviously, a strong specialization
fields for each of them, but you still reach bottlenecks in the
production when you have a few guys who understand database
programming, or you have a few designers who understand social design.
So even with a throughput of 140 people, it doesn’t mean that we can do
everything and have it go through as fast. So we have to sort of pick
and choose.



Some of the features that have been requested, or need some loving,
theoretically, we have enough people to work on alongside everything
else, but there are bottlenecks in the production chain where things
fall apart. What we are doing, however, is we’re taking our focus off
these features, in general, to refactoring older features. This is a
process that started, basically, at the beginning of this year and will
continue throughout next year till the point where we’ll halt
production on new features, and turn to refactoring the old features or
updating older game designs.



You can call it development eras if you want. So you add a bunch of new
features, then you reiterate on those features until you’re satisfied
that they’re polished. In general, what you see is, and I guess the
players have been seeing this, and they will be seeing it more in the
next release and the next release thereafter is a focus shift to
iteration.

Ten Ton Hammer: So that
ties into the final question. What’s next?
style="font-weight: bold;">


Arnar Gylfason:
Well, we’ve been working on an the Incarna system for
years. Four years, to be exact (stop laughing). We feel that we’ve
reached a critical mass in development there, and that’s in full
production. That’s something to be looking for very shortly. I wouldn’t
want to commit to it for this expansion and I won’t commit to it for
this expansion, but moving into next year, absolutely, you’ll be seeing
some amazing things on the Incarna front.



Then you’ll have the Incursion feature, or the Invasion feature. We’re
iterating on the planetary interaction feature as well that we launched
earlier this year, and we have plans on iterating on the combat and
tactical UIs at some point next year.



That’s the plan, but plans can sometimes change. Plans will change, but
currently our plan is that we want to iterate and update the tactical
UI for fleet fights. You can always fix live for this particular set of
problems, but having to render hundreds or thousands of ships onscreen,
or we can take it down a notch and say, when you’re in a fleet fight
this big, perhaps you need a different view. You just need to render it
different. So that’s something we want to move into and have a certain
level of detail loading of combat views based upon the magnitude of the
battle you’re in. That’s a sort of game design change that can help lag
a lot in the future, but it’s a fairly big refactoring of "Excel in
Space." I don’t think players want to lose their "Excel in Space."