Hacking EverQuest Part Three - SOE's Response
One of our series we saw some of the reported examples of the
exploits that are taking place in EverQuest,
primarily on the Time Locked Progression Servers. In Part
Two we spoke with two individuals who had experience in
limiting the ability to hack on their emulated servers. In our final
part today we approach Sony Online Entertainment to find out what can
be done from their end to put a stop to the hacking.
We had determined that the early fixes that Rogean and Haynar had implemented on their emulated servers may not be ideal on a large commercial scale for SOE. However, since the time of the interview, Rogean did contact me with some updates.
"Using packets with unpredictable variables to crash a client running MacroQuest was just the beginning for us," Rogean began. "Even selectively using that method, many of the hack programmers found new ways to 'toughen' their distributions to make them impervious to the sort of vulnerabilities that we exploited to detect them... We came up with a brand new system of detection that we were able to embed into the client directly without actually modifying the client itself.
Through this system we can then keep a watchful eye on the memory space of the client. We can check to make sure that there are no external modules loaded into the client that are reading or changing the client's memory. If this system detects this, it sends the name of that module with a specific flag to the server, which logs it for later review. All of these flags came back as MQ2Main.DLL on Project 1999. This system introduces a whole new level of detection available to us that can also be adapted to the hackers' attempts to avoid it. They may try to avoid it, but they are taking the chance that we alter our detection to pick up on them and they lose their main character. In most cases, this is not worth the risk."
The bigger issue that SOE may face is that once the cheaters are comfortable using the hacks and do not see punitive actions taking place against the hackers, they feel all the more confident that they can get away with using them.
"It's just like if in real life," Rogean continued. "If you don't see many police pulling over speeders or you get away with driving faster and faster over time, you get comfortable driving those speeds... until finally you get pulled over.
"That's what we did. We pulled over 365 people at once, publicly, and that will stick in everyone's minds that we take our anti-hacking rules very seriously, and we won't give up trying to detect them."
I called up Thom Terazzas, Producer of EverQuest. If anyone knew what SOE was doing or would be doing to combat the hacks, it would be him. Thom was familiar with the problem. He also knew the importance in letting players know that bans are happening.
When we spoke about customer service vocalizing the bans Thom recognized that it hasn't been happening much lately.
"I don't think they've done it recently as far as notifying the community about mass bannings," he said, "but it is a big deal to us too... I think what we need is just a little more promotion of information from CS a little more often on the message forums--just to let them know we're on to them."
Even though it is a concern to Thom and his development team, it may take some time yet before the issue can be fully addressed.
"Obviously there's a lot of juggling we need to do in order to make it a priority," he continued. "The expansion, [Veil of Alaris], is something that we're really focused on right now. That has gotten, I would say, 80 percent of the[development] focus here. So doing anything that dramatically combats the hacking is something that we would really like to do, but it is not something we've been able to do."
That's not to say, however, that no time has been spent investigating the cheats.
"When the dev team finds any extra bit of time and are able to do a little more logging and checking, we do it," Thom assured me. "We caught quite a few people warping and we checked times and locations of where they were standing when particular mobs spawned and how long it took for them to die. Customer service has been following up on this. So we have been helping CS do their work by checking that type of information. With these new checks over the past few months we've actually caught a lot of those guys and nailed them to the wall."
But what about the theory that SOE doesn't want to stop hackers as it would mean fewer paying accounts, thus less revenue? I asked Thom about this.
"Wow," Thom responded. "I think that's pretty far-fetched. I have never heard anyone here ever say anything like that," he continued emphatically. "There's never any context of money in association with not fixing an issue. We should not make any business decision based on what other people are negatively trying to do. We should go about our business in a positive way and take care of these people. Obviously that's what we have to do--a higher effort. It should just be put at that higher priority.
"Believe me. I've never heard anyone say or even suggest that we should not do anything about the hacks. That's never been said in any conversation I've been around."
"I know that we're somewhat limited on what they're able to report or record," he answered. "I know there's not much of a mechanism for them to show proof of what's happening. The /report command will capture text," but he also agreed that even that is somewhat limited in actually capturing evidence of cheats.
"But maybe we can do something simpler or quicker that will help in detecting the cheats even from the other players' perspective and allow us to flag people for audit. I don't know what that would be, but it's definitely worth a conversation with my team."
In the days following our conversation, Thom indeed did follow up and spoke to the code team.
"We've decided this is definitely something that needs attention as soon as we get through the fog of the expansion," he told me. "It's probably not going to happen right now, but I did talk to CS as well. They have been diligent on taking the reports and putting a watchdog eye on people that need it.
"Customer Service does want reports of suspected cheating," he expanded. "They've asked me to convey that they want the continued /reports followed up by tickets on those suspected of cheating with as much detail as possible. The more detail the better."
As is typically the case in MMOGs, Thom also noted that CS probably wouldn't follow up with a reply to the ticket when the cheater has been caught, but they do appreciate and use the reports and petitions to monitor the user's activity. "They do review every ticket," he said.
"The /report command will log the last lines of text. I think it goes back up to 100 lines or so. So if there's any communication players witness in the various channels, that command will help get some definite timelines and locations on when the event occurred."
My discussions with Thom led me to believe that there is still a lot of passion behind the developer curtain of this 12-year-old game. Resources may not be at their peak and the downside to that means these issues may continue for a while yet before a major focus in development can be shifted towards combating the hacks. However, the community itself is also very driven. With a combined effort from the community perhaps some of this can be satiated until more resources are available from SOE to supply more tools not just to the CS staff, but also to the community to help flag the offenders.
Hacking and cheating are cancerous to any online game as it directly affects other players. No gamer and no developer wants hacks to be happening in their game. The reality of it though, is that cheaters cheat, and likely always will. It takes effort from everyone to let them know they're not welcome in the games we play. We, as community members can speak out against them and get the information to the Customer Service staff. The developers can take their role by supplying CS and the players with the tools they need to uncover the hacks. And maybe, just maybe, with enough drive and enough people from the community and the development team acting together, these cheaters will no longer feel welcome, nor safe, in our games.