BlackBerry's outage caused by huge e-mail backup

By Lance Ulanoff, Mashable October 12, 2011 -- Updated 2233 GMT (0633 HKT) | Filed under: Mobile

BlackBerry customers today started complaining on Twitter of mail delays and inaccessibly on their BlackBerry devices.

(Mashable) -- In a brief press conference Wednesday, Research in Motion CTO for Software David Yach said a backlog of messages to Europe has created a cascading outage effect for BlackBerry users around the world.

The company has determined the root cause of the initial European BlackBerry e-mail service outage, which started three days ago, and said there's no evidence that a hack or security breach was involved.

RIM customers in Europe have been suffering from major outages for days, but it wasn't until today that the Americas caught the bug and BlackBerry customers started complaining on Twitter of mail delays and inaccessibly on their BlackBerry devices.

Yach described the initial outage as a failure of one of RIM's core switches. However, the real trouble began when RIM's redundant systems failed, as well. "The failover did not function as expected," Yach said, "despite the fact that we regularly test failover systems." This led to a significant backup of mail.

RIM responded by throttling service in the impacted area to stabilize service, which apparently resulted in backup of mail in other regions trying to reach RIM's European customers. The Canadian-based company would not reveal the exact number of RIM customers affected, saying only that many customers are impacted in a variety of ways. "Some see delays, some service interruption," Yach said.

For now, RIM says it is working "around the clock" and "globally" to try and restore service. It has not throttled service in regions outside Europe. RIM's Yach said they're focusing on clearing out the backlog of e-mail, much of it in Europe and for anyone trying to deliver email to Europe.

When asked if RIM might try and wipe the slate clean and clear out the backlog without delivering messages, Yach answered swiftly: "All email will be delivered. We will not be dropping any email messages."

While Yach said RIM knows the root cause of the initial failure, it wasn't willing to share it — at least until it finishes its own thorough investigation. That won't come until after all service is stabilized and restored.