| Comments

The Day the Internets Broke

On Tuesday, a bug in the OpenSSL library was disclosed. It even got a name - Heartbleed - and a logo. And it turned out to be as severe as it can get.

OpenSSL is a library widely used on the internet to secure communications. When you access a page via https, when mailservers communicate securely with each other, when remote servers are accessed by sysadmins, OpenSSL encrypts the communication so no third party can listen to your private transactions. A bug in this library is bad enough by itself. But this one went a step further, or as security expert Bruce Schneier puts it:

“Catastrophic” is the right word. On the scale of 1 to 10, this is an 11.

The bug didn’t only allow attackers to listen to communication. Instead, it would allow random chunks of memory to leak to an attacker. Why is this particularly bad?

  1. Neither as user nor as an admin of a system would you see that you’re being attacked.
  2. This leaked memory could contain passwords, usernames, sensitive data or even worse: the keys used to establish the encrypted connection, opening the door for other attacks, like listening to communications or decrypting past communications.

How was Meltwater affected?

Immediately after we received a notification from security mailing lists and our hosting partners, we worked under the assumption that all of our resources were potentially compromised and checked all systems.

In particular, we first checked our CRM system, which holds information about our customers. This wasn’t affected, as it uses a different library.

Then we checked all our products – app.meltwaterbuzz.com, service.meltwaternews.com, app.meltwaterpress.com, icerocket.com and likealyzer.com. None of those were affected by the bug. They run either a different library or a different version of OpenSSL.

The only vulnerable page we had was our corporate homepage, www.meltwater.com. This page uses a different SSL certificate than our other services, so even if the encryption key has leaked while we were vulnerable, no other systems can be compromised. A lot of customers use this page to log into mnews, but those passwords never make it into memory of www.meltwater.com, but are passed through to the mnews service directly.

As a preventive measure, because security is of paramount importance and, especially in cases like this, we’re justified to be overly paranoid, we’re putting on our tinfoil hats and generating new certificates for all public systems.

We take security seriously at Meltwater. We want our customers’ data to be as safe as possible. Our servers run Red Hat Enterprise Linux, which is rigorous about security testing, and we patch our servers once a month to apply the latest security fixes.

Nevertheless, software is written by humans, and humans make mistakes. We don’t blame the developers of the OpenSSL library for this bug. We are glad we weren’t affected and didn’t lose any data – especially none of our customers data, which we are obliged to keep extra safe.

While Meltwater’s systems were safe, many services on the internet were impacted. We suggest you change passwords for all sensitive services you are using. More information on the bug from a user side can be found at heartbleed.com, and The Register has a good technical explanation.