EVE Online at Ten Ton Hammer - Taking MMOG Gamers to the Next Level!

Last year Ten Ton Hammer awarded EVE Online as the Best Community in a MMOG of the year. Why? Because the community knows what it wants and has had a fundamental part in designing the future of the game. Ten Ton Hammer caught up with Arnar Gylfason, Senior Producer of EVE at gamescom 2010 to address some concerns that have been going about the community. Lag, features backlog, and the future of the game are our topics in this exclusive EVE Online interview from gamescom 2010.

---

Ten Ton Hammer: The EVE community has brought up concerns with the performance and the lag with the most recent expansion. Can you comment on this?

Arnar Gylfason: Actually, this has come up a lot in recent interviews, and naturally so. This is of concern to the community and a concern to the developers as well.

I’m not sure that you’re familiar with the agile Scrum method of making computer games. Basically, it’s teams of people working in two week sprints. We have people who are extremely specialized in the networking layer of the code and core mechanics of the code and would be the perfect people to work on these problems. However, EVE is a complex game and we have ten years of development code to try to figure out where the bottlenecks are and what the problem is. It’s not something where you tell people: "you have two weeks; fix it!" That’s just not going to happen.


So we told these people, who were on scrum or other teams making features or other stuff: "here’s a list for you guys. You have to set aside time, each with your own strengths, to prod, investigate, and research what the code is doing and why."

I call this epiphany development. You're working on a specific problem. You sort of know what it is. You have to get all sorts of data on it; you have to get reproduction on the problem. It’s all sort of iffy and hazy, then one day, you have an epiphany on what the problem is and how to fix it. Then actually fixing it takes a day of just coding, peer reviewing the code, and testing. Then you’re done. But the process leading up to that can take days or weeks or even months of investigating, painful reconstruction of scenarios, and it often entails a great deal of community interaction.

We’ve held mass tests where we get players on our test servers, hundreds and hundreds of players, to recreate fleet battles to see why this occurs in fleet battles. Why doesn’t this happen to all of these people fighting NPCs all at the same time? What’s going on here?

So, that was the set up that we had and it was working fairly well. We were hammering out some fixes all of the time, but they weren’t groundbreaking fixes to the lag. I shouldn’t say 'fixes' because we’ll never fix lag; we’ll just up the limit on what is acceptable, because when we fix lag for 1000 people, then they’ll just bring in 1200 people. If we fix that, then it’ll be 2000 people. So, incrementally upping the limit on the number of people who can be in fleet battles is something we’ve been doing for a while.

Now, something broke somewhere along the way, probably in the Dominion expansion or shortly after the Dominion expansion, where the performance just decreased all over. We had these people working on it in this fashion since then, and they’ve been investigating potential causes. They’ve been fixing other problems. They find a lot of side-line problems when they’re investigating these problems. So, they’ve been fixing that and publishing that.

Now, obviously, the problem has reached critical mass. The performance is not up to snuff on what we consider a playable experience for our fleet battles. The developers themselves are getting more frustrated that they can’t figure this out, or they feel sometimes that even if they time set aside to do this, it’s not enough. So now, we’ve formalized this team. We’ve taken pretty much the most brilliant minds and clustered the technology and network layering technology in programming and put them together into one team and it’s sort of a skunkworks team. They are allowed to do pretty much what they want and try theories that they want. We will facilitate them getting it out and getting it tested. The combined experience is probably forty, fifty years in high-end clustering technology application programming.

We are hoping, and like I said, we can’t give an exact time quote. I can’t say that yes, in two weeks, we’ll fix lag because we have all of these people working on it. But we do have all the people at CCP who could be working on it are working on it dedicated 100 percent, and we have teams of support personnel around them making sure that they have everything they need, that they have the test servers they need, that they have the reproduction stuff they need. We even have a separate team building what we call thin clients, which is a client that runs on a low CPU, low memory, and low ram, so that each computer can run tens or hundreds of clients. We can recreate these fleet battles by scripting these clients to behave as players would be running actual clients. So, we’re able to basically make fleet fight in a can running a small network of ten or twenty computers.

So, there’s a lot of effort going into this, not just for the five or six people who are actively working on identifying where the problem lies and how to fix it, but there is a whole team of developers making support tools for them. Then there’s operational staff, customer support staff, quality assurance staff, and community staff that are supporting them. In all, we have about twenty-five people working on this now.