Ordinn posts about Realm Issues

Ordinn over at Blizzard has posted the following in the Blizzard Boards about recent Realm Issues:

The overwhelming success of World of Warcraft has brought hundreds of thousands of people together to adventure in Azeroth, and concurrency numbers are well beyond what we expected or even hoped for. Unfortunately, this high concurrency, especially when concentrated on a small number of realms, initially caused issues with our hardware infrastructure. We were able to streamline our code to increase performance in the weeks following launch. However, the holiday season nearly doubled our player base, and it quickly became apparent that in order to handle not only the current player base, but all future players as well, we needed to make some upgrades to our infrastructure.

Last Thursday we made our first such upgrade. 20 of our 88 realms were moved off of the original hardware and placed on a new hardware configuration. These 20 servers initially performed very well, up until we reached our maximum concurrency Friday evening. The high population numbers uncovered an issue in the new backend shared infrastructure. This issue caused some players to experience severe lag and disconnects on a few of the realms, making them virtually unplayable.

In order to stabilize the affected realms and allow as many players as possible the ability to continue playing, we lowered the population caps by 30%. This stabilized the realms to the point where 70% of the players on the realms in question could play, but it also resulted in large queues.

The problems were attributed to high concurrency numbers on individual realms putting extreme stress on the backend infrastructure. We were able to address this problem by implementing additional hardware into the infrastructure this afternoon. This additional hardware has allowed us to stabilize the affected realms, and thus increase the server caps. We will continue to monitor the performance throughout the evening. If we notice any of the performance issues starting up again we will lower the population cap level enough to stabilize performance.

We sincerely apologize for the inconvenience this caused our players this weekend. This process coincides with our constant efforts to improve the current performance of World of Warcraft, and sometimes issues can arise when implementing these improvements. We will do our best to prevent similar situations from happening in the future, and we once again thank you for your patience and understanding.