Heart-not-so-bleed!

Executive overview

Cyphertite was not vulnerable to the Heartbleed attack.

Management overview

Disclaimer: These findings are true for the Cyphertite service ONLY. We are not making any statement about other people’s results and/or attack surface.

We did not receive an advance warning of the Heartbleed vulnerability. As soon as the news broke the Cyphertite team shut down all services and began investigating its exposure. We quickly established that the website was not at all vulnerable to Heartbleed (it uses OpenSSL 1.0.0f) but we used a vulnerable version of OpenSSL on the service itself (OpenSSL 1.0.1c). The service was patched and brought back online with 3 hours of the Heartbleed announcement, however, at that time, we did not know the extent of Cyphertite’s exposure.

In order to make a definitive public statement we spent days attacking Cyphertite to see what this bug meant for us. First we spent time determining if there was an attack vector. Once we had that established, we went down the rabbit hole to see if we could execute an attack. The simplified attack vector is as follows:

1. Attack Cyphertite service with Heartbleed
2. Obtain hashed password
3. Obtain client certificates
4. Obtain Cyphertite server private key
5. Launch MITM (Man-in-the-Middle) attack with server key material
6. Attack Cyphertite client with Heartbleed
7. Obtain Cyphertite client AES key
8. Obtain Cyphertite client HMAC key
9. Decrypt individual chunks and/or CT files from client

As improbable and unlikely success of such an attack is, we decided that we owed our customers nothing less than to verify that it did not, and could not happen.

The initial attack failed since we could glean nothing at all from Heartbleed and we had to modify the attack by obtaining a certificate bundle which requires the creation of an account. This was good news because that meant we never leaked hashed passwords. Note that passwords are only used to log onto the website and the service; passwords are NEVER used for anything else. That said if we would have leaked a hashed password that means an attacker could have obtained someone else’s certificate bundle. Even though that sounds bad, the only risk there is that one would be able to read or write encrypted chunks. Since the chunks are addressed by a digest one would have to start guessing values making obtaining chunks highly improbable. Assuming one would obtain chunks they would still be worthless without keys.

The modified attack was to create a Cyphertite account and use the certificate bundle and hashed password to get past login. At this point we attacked the service in order to see if we could obtain anything of value. We used the outstanding code from Filippo Valsorda on GitHub.  We modified the code to run the attack continuously and search for fragments of key material in heartbeat responses from the service. We searched for DER and PEM data. We ran the attack millions of times over the course of several days and the good news is we came always out empty. Of course the replies from the server were also analyzed by humans to ascertain that we didn’t miss anything.

During this testing we found out that in our particular environment the attack could only return up to 32 KB of server memory. The reason we could not get the advertised 64 KB is that on OpenBSD we were hitting guard pages causing the connection process to dump core, i.e. crash.

Additionally we investigated our logs looking for “smoking gun” core files and other suspicious activity. That search came out empty as well. We are therefore comfortable stating that Cyphertite was not vulnerable to the Heartbleed bug and there is no evidence it ever came under attack.

With this information we can ascertain that the attack can not proceed past step 4.

Additional details
The Cyphertite cloud service sits on the internet listening for connections. This particular code is internally known as cr (“chunk router”). When a client connects with ct (Cyphertite) the first thing that happens is that cr forks the process. The reason we made that design is twofold, first it gives us greater parallelism and second, it provides resource separation between individual connections. This also means that Heartbleed CANNOT be used to attack other active connections. In the Heartbleed case this means that the ONLY interesting piece of information available to that process are the cr certificate private keys. While investigating memory maps of cr we easily, as expected, found the DER and PEM key material. When investigating this bug we went through the memory map looking for other potential juicy bits and found nothing. With this information in hand we determined the attack vector and what the prerequisite was for a successful attack.

The next step was to modify the attack tool get past certificate exchange in order to effectively attack the service. While modifying the tool we added several options in order to aid in the search of key material. Working with large keys by hand is error prone so we added the option to read in the server keys directly from a certificate. We also added the option to modulate the size of the returned buffer and number of bytes of key material to search for. This enabled us to experiment quickly while being certain that there were no human induced errors.

The key search algorithm is pretty simple: look for N consecutive bytes of both the PEM and DER in every returned heartbeat buffer. N is variable and the entire key is divided by that number and the individual key segments are searched for. This means that we looked for fragments of both entire keys every single time. We played with the settings here and found that at most we’ve ever seen is 3 bytes that matched. Looking at those instances we could determine, in context, that it was not actual key material but a “lucky” hit.

Over the weekend some additional information came to light about leaking P, Q and other details.  It might be worth mentioning that Cyphertite uses ECC (Elliptic Curve Cryptography) for TLS but we went ahead and went through various attack sequences posted on the Internet.  Those methods also came up empty.  To be explicit, the only thing we have been able to “leak” in the case of Cyphertite is the client side certificate.  This is the certificate that the attacker must use to even commence the attack.  The leaked buffers were always composed the same way.  First came the payload we sent to the server followed by the client certificate followed by zeros.  Thus we can ascertain that there simply was no valuable data to analyze.

We used the tool against cr machines and mocked up tools for several days. We ran millions of iterations of the Heartbleed attack and found nothing interesting at all.

Conclusion
This is by far the worst security bug that the Cyphertite team has ever seen. We are glad that our security paranoia paid off when it was needed the most. We were even mocked by some about this paranoia and to that we simply say: told you so! This proves again that one can never have too many layers of security and process. It also comes to show that having all infrastructure under your own control is the only way to ensure customers safety. If we had 3rd party machines/cloud or what-have-you we would not have been able to effectively attack our own service and obtain this type of information; not to mention less or no control over things such as physical access to machines, upgrade schedules, keying material, etc.

Stay safe, use Cyphertite! We got your back!