Sealing the Rift: A Look at Account Security Featuring Exclusive Interviews with Scott Hartsman
In all the myriad tasks associated with gaming, none can rival launching an MMOG in terms of scope and magnitude. As the genre continues to evolve and grow the difficulty level increases exponentially. In addition to higher expectation levels and higher populations, new mechanics and systems keep getting added to the mix which creates a perfect breeding ground for launch day snafus. When Rift experienced a remarkably smooth launch that was marred only by server queues created by higher than expected popularity, it seemed they had dodged the pitfalls that beseech most of their brethren.
But early on the reports began pouring in of account compromises, hacked accounts stripped of all their valuables and left like some stolen Honda on the side of the road. Our very first post-launch interview, which took place during our trip to GDC, hinted strongly at just how much of an issue that account security would become. Our first inklings of the issue were the overabundance of gold spammers that set up shop within hours of the live launch. We asked Scott Hartsman about this:
Ten Ton Hammer: Were you surprised by the amount of gold spammers that were present at launch?
Scott: We really werenÂt. We had a chat filter installed and listening to all that was going on throughout beta and it was already paying attention to user reports. After about four days of internally logging this information and looking at what users were reporting, we turned it on and began the process of kicking spammers offline and blocking their access. So the first three days were really loud, we didnÂt want to jump the gun on auto kicking and auto banning, but once we knew it was working properly it got a hell of a lot quieter as we started banning accounts.
Scott then proceeded to show us some pretty amazing integration he had on his tablet computer that allowed him to remote access any realm, view spam filters and access detailed information. Color us impressed. As we viewed the spam filter you could see how it compiled its own dictionary and flagged words based on how users reacted to them. Using this info it began to cast a net with which to catch the offending users and remove them from the game.
Scott: ItÂs hilarious to see some of the unique and creative ways the spammers are using to try and get around the filters, but the users arenÂt fooled and they report it just as quickly. There were some gold farmers that woke up to a really bad morning this morning.
Please, give me back my armor.
Scott: You have to be, the speed with which these attacks come is truly surprising. It shocked me and I have been doing this a long time. The day we opened for headstart, the incoming traffic from people trying dictionary attacks on peopleÂs accounts was unreal.
Ten Ton Hammer: Do they try DDoS attacks on you as well?
Scott: We have load balancers and firewalls to deter that, but one of the neat things we are doing for people trying to attack us with bad logins is that if you do it enough times we redirect you to the Chinese Internet Ministry to report yourself. We do that in the hardware so we donÂt have to write special software for that. Fortunately internet security hardware has really leveled up in the past few years to stop brute-force style attacks.
Despite assurances to the contrary, this probably isn't safe and secure.
As the second week gave way to the third the number of thread on the forums related to account hacks and security exploded. While many users wondered just what the hell was going on, the community seemed to be divided firmly into two camps; those who had been hacked but swore they didnÂt do anything to jeopardize their accounts and those that ridiculed them. When we caught back up with Scott at PAX East, it was once again at the forefront of our discussion.
"IÂm primarily concerned with keeping our customers safe. ThatÂs the overriding factor, the fact that there is active fraud and active theft and active breaking the law against our community and it pisses me off. "
Ten Ton Hammer: Do you feel using E-mail address for logins is a security compromise?
Scott: It really isnÂt. The two main ways that people will gain access this way is from either an account that was previously compromised in another game and the same credentials are used, or from phishing websites . So if you used the same credentials in both games it wouldnÂt matter if it were a unique user name or an email address, they will still have that information. The newest phishing site we came across was riftgame.net, which the link said riftgame.com, but you know the nature of links and how they work Â it took you to the .net site and they phished out a bunch of passwords that way. We have been reminding people ÂdonÂt click on links in emails, always copy and paste it Â anything that will come from us will allow you to copy and paste.Â
Ten Ton Hammer: Do you have a legal department that handles the phishing sites?
Scott: Yep, exactly. Not only that but then we get them added to the search engines as known malware sites. So we are working with ISPs, search engines and trademark people to combat these things.
Ten Ton Hammer: Are the brute-force protections still in place?
Scott: Yes, if you try and brute force us, and you come from China, you will be redirected to Chinese Internet Protection website and you will be submitting a report against yourself. That being said security is still something we are working on day by day, improvement by improvement.
Ton Ten Hammer: What are you working on right now?
Scott: What we are working on right now is economy locking your account, such as if you log on from a new IP address that we donÂt recognize from you and you havenÂt authenticated yourself by answering the secret question or some other form of verification, your character will not be able to give, sell or destroy anything until you look in your email and get an unlock code. We are also looking at a two-factor, like an iOS or Android app, which we are pushing forward on as much as we can.
Ten Ton Hammer: One of the major gold selling sites stated that Rift was the most requested game at launch in the history of their business; does this flatter you or frighten you?
Scott: Flattering, since you know there is always going to be gold sellers, which is why we launched with a chat filter. We are currently working on taking the next step with that chat filter and applying it to in-game mail.
Ten Ton Hammer: How many accounts have you banned since launch?
Scott: Off the top of my head, itÂs multiple thousands. I donÂt have the hard number in front of me but we get reports every hour of active hackers. The real problem is that a lot of them are registering accounts with stolen credit cards and a lot of them unfortunately recycle credit cards of people who buy from them. ItÂs really dangerous to give your credit card to those kinds of places Â itÂs got to be a hellacious experience to wake up one morning and find that the company you just bought gold from used your credit card to charge 250 copies of the game to. ItÂs best just to stay away from that whole ecosystem.
Ten Ton Hammer: Is there any way to stop it completely?
Scott: I think that economy locking will be interesting. Economy locking and the cell phone based two factor will be a good place to be. When my cell phone becomes my account key then IÂm pretty confident that itÂs only me that is getting into that account. I know it isnÂt the cutting edge of technology but it is extremely effective and among the least inconvenient ways to help secure an account. IÂm less concerned about stopping the gold selling market; itÂs always going to exist some way or another, and IÂm primarily concerned with keeping our customers safe. ThatÂs the overriding factor, the fact that there is active fraud and active theft and active breaking the law against our community and it pisses me off.
As the epidemic reached its crescendo, the mystery of the pervasive account compromises was finally solved Â and it was done by a player who was doing a bit of ethical hacking in an attempt to help fix the problem. Known on the forums as ManWitDaPlan, this user was frustrated by his own experience of loss but noticed some discrepancies with a typical account hack:
With the hacking fiasco currently raging across the game, I'm seeing enough strangeness to wonder if Trion isn't facing two simultaneous issues: a wildly successful large-scale hacking of the game's security and a potentially catastrophic problem with the game's back-end code. My particular "hacking" case deviates from the norm for game account hacks - my inventory is mostly gone, as is my money, but my bank was never touched.
A number of other posts on the forums include behaviors that are inconsistent with account theft: partial inventory losses, inventories left alone but banks are emptied, banks left alone but inventories are emptied, etc. etc. etc. Also, quite a number of victims that report unorthodox account damage were hard targets to begin with and tested clean after-the-fact.
I suspect, and I know this will probably not be confirmed, that the hacking was a bruteforce attack against Trion's account management and/or authentication servers, and/or an active exploit against one or more weaknesses in Trion's systems, and not any form of client-side malware. This would certainly explain why people that are more than savvy enough to not get nailed by keyloggers, etc. still got hit. This particular concern is a major catastrophe for Trion if it's true, for it calls into question the entire security side of Trion's operation and throws loads of obstacles between Trion and making money from game account subscriptions. Needless to say, Trion is likely desperate to get the hacks under control, whatever the attack vector may be, before significant numbers of players withdraw their billing data and opt out of continued play come April.
I also suspect, and this too will probably not be confirmed, that Trion is also fighting a back-end data-corruption issue that is the cause of the stranger "hack" cases where someone that had a reasonably uncrackable password fell prey to, and I suspect this may relate to either the hotbar-icon hotfix, or the attacks against Trion, or both. A corruption issue would certainly cause abnormal-for-a-hacked-account changes to inventory/bank contents, and could also explain cases where one of a number of characters on one account was hit and not all of them, random characters were deleted, etc., which is also deviating from the norm a bit for hacking cases. Distinguishing such an event from a hackfest would be tough at best and at worst well nigh impossible, and it'd be easy for an inventory bug to get lost in the sea of stolen inventory items during the hacking spree.
Trion is stuck in a no-win situation at this point, as each passing hour waiting for (surprised and subsequently hideously overloaded)Trion's customer support adds more frustration to players waiting for support to fix their destroyed characters (I'm on day three since posting a GM request on my "hack"), and each new report of unusual circumstances that don't fit the normal modus operandi for a hacked account serves to only undermine any faith in Trion's account - and by extension, billing - security. This came on hard and fast and Trion is obviously scrambling to stop the ongoing hacks, shore up security, and perform some semblance of damage control. In the meantime, victims of the security failure that shouldn't have been victims under normal circumstances are already looking to divest themselves of this particular Pandora's Box before it evolves into a bigger problem.
To make matters worse, some folks are reporting being told it may be several days before their destroyed accounts can be recovered. Many of them are not going to wait that long, and when they go, the $10-$15 a month they were planning to spend goes with them. This means even more pressure for Trion to get a handle on the situation, and fast.
While a post like this may be glossed over by some companies, it seems that the folks at Trion were noticing something similar and made the unusual move of directly contacting this user to find out more and work with his data to craft a solution. The result was a hotfix that was developed and deployed in just over two hours from the time of that conversation. Scott Hartsman publicly thanked ManWitDaPlan for his assistance but also warned that account security was Âa multifaceted issueÂ and one that required constant vigilance on the part of everyone. The total number of compromised accounts was only 1% of all accounts, but with a game as popular as Rift, that can be a significant number indeed.
One code to unlock them all.
We would like to thank Scott for sitting down with us on both occasions and we look forward to many more interviews with the entire Rift team in the future.