EVE Online Stackless Python Upgrade Q&A
Questions by Cody “Micajah” Bye
Kristján “porkbelly” Jónsson, Senior Software Engineer for CCP, during one of the company's "Formal Wednesdays".
A few weeks ago, CCP Games issued a press release that announced their successful upgrade of Stackless Python and that this upgrade would increase the speed of the gameplay for EVE Online players. Many players of EVE Online probably shrugged their shoulders and continued playing their game, noticing the performance increase but generally not thinking twice about the upgrade.
However, the staff at Ten Ton Hammer (we were in San Diego at the time) found the press release to be interesting, albeit a bit short for our tastes. We wanted to know more, what had actually occurred in this update and why was it necessary? What is Stackless Python? Does anyone else use it besides CCP?
These were our questions, and (thanks to Valerie Massey) we were able to send those questions to Kristján “porkbelly” Jónsson, Senior Software Engineer for CCP. After you read the Q&A, you’ll know that Kristján knows his tech and all of his answers are both informative and honest.
Ten Ton Hammer: To start, could you summarize what your high-level programming language, Stackless Python, accomplishes within the EVE Online world? How does thread-based programming work? In your press releases you don’t offer much of an explanation on Stackless Python and I’m sure the community would be interested in hearing a simplified version of what Stackless Python is all about.
Kristján: Python is by now a well-established programming language. Being an interpreted dynamic language, it allows for great expressivity to the programmer and rapid development. This makes it easier to focus on the software architecture and the problems at hand rather than having to deal with the infrastructure of languages such as C++. In addition, the dynamic nature allows a very high turnaround rate. We can make changes to the code and see the effects in the running client or server without restarting them. It turns out that performance in terms of programming output by developers is just as important as the performance of the product they are developing.
Stackless adds a further twist to this by allowing tasks to be separated into smaller tasks, Tasklets, which can then be split off the main program to execute on their own. This can be used for fire-and-forget tasks, like sending off an email, or dispatching an event, or for IO operations, e.g. sending and receiving network packets. One tasklet waits for a packet from the network while others continue running the game loop.
It is in some ways like threads, but is non-preemptive and explicitly scheduled, so there are fewer issues with synchronization. Also, switching between tasklets is much faster than thread switching, and you can have a huge number of active tasklets whereas the number of threads is severely limited by the computer hardware.
EVE Online runs well on many different types of machines, but upgrading Stackless Python made it even faster.
Ten Ton Hammer: What sort of factors helped push you towards the decision to upgrade Stackless Python? You mentioned some upgrades to Python and the NeedForSpeed sprint, but why choose to upgrade the client now rather than before Revelations II or farther down the road?
Kristján: Due to various issues, mostly being swamped with day-to-day development and maintenance of EVE, we didn’t pay enough attention with keeping the Python part of the engine up-to-date. At various points we have attempted to catch up but we never had the breathing space to do so. This time we had the core technology ready and decided to go for it. It was a question of reserving a space in the release schedule for it. We wanted to apply the change in the breathing room after Revelations II when the initial flurry of hotfixes had subsided. The patch went out without any other change so that we could monitor the effects of this change in isolation.
Ten Ton Hammer: Honestly, the performance improvements with Stackless Python seem to be pretty substantial and certainly frees up some resources from the users that you could take advantage of. Might we see some upgrades to the EVE client down the line, like improved graphics, more intensive AI, and other enhancements that would not have been possible without Stackless Python 2.5?
Kristján: Improvements to our graphics are something we’ve been working on for quite a while now and players will begin seeing those this fall. However, some of the slack that is generated by improvements such as these is usually taken up somewhat by something else. On the client, for instance, the frame rate may go up to compensate. On the server side, we have found that increased responsiveness reduces lag, which again stimulates the users to place additional load on the servers. People may be hesitant to give up the improved responsiveness for more features. It’s a question of balance, I suppose.