Heartbleed

For other uses, see Heartbleed (disambiguation).

Logo representing Heartbleed. Security company Codenomicon gave Heartbleed both a name and a logo, contributing to public awareness of the issue.

Heartbleed is a security bug disclosed in April 2014 in the OpenSSL cryptography library, which is a widely used implementation of the Transport Layer Security (TLS) protocol. Heartbleed may be exploited regardless of whether the party using a vulnerable OpenSSL instance for TLS is a server or a client. It results from improper input validation (due to a missing bounds check) in the implementation of the TLS heartbeat extension, thus the bug's name derives from "heartbeat". The vulnerability is classified as a buffer over-read, a situation where more data can be read than should be allowed.

Heartbleed is registered in the Common Vulnerabilities and Exposures system as CVE-2014-0160. The federal Canadian Cyber Incident Response Centre issued a security bulletin advising system administrators about the bug. A fixed version of OpenSSL was released on April 7, 2014, on the same day Heartbleed was publicly disclosed.

At the time of disclosure, some 17% (around half a million) of the Internet's secure web servers certified by trusted authorities were believed to be vulnerable to the attack, allowing theft of the servers' private keys and users' session cookies and passwords. The Electronic Frontier Foundation, Ars Technica, and Bruce Schneier all deemed the Heartbleed bug "catastrophic". Forbes cybersecurity columnist Joseph Steinberg wrote, "Some might argue that [Heartbleed] is the worst vulnerability found (at least in terms of its potential impact) since commercial traffic began to flow on the Internet."

A British Cabinet spokesman recommended that "People should take advice on changing passwords from the websites they use... Most websites have corrected the bug and are best placed to advise what action, if any, people need to take." On the day of disclosure, the Tor Project advised anyone seeking "strong anonymity or privacy on the Internet" to "stay away from the Internet entirely for the next few days while things settle."

As of May 20, 2014, 1.5% of the 800,000 most popular TLS-enabled websites were still vulnerable to Heartbleed.

TLS implementations other than OpenSSL, such as GnuTLS, Mozilla's Network Security Services, and the TLS stack used by Microsoft products, were not affected because the defect existed in the OpenSSL's implementation of TLS rather than in the Internet protocol itself.

History

The Heartbeat Extension for the Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS) protocols was proposed as a standard in February 2012 by RFC 6520. It provides a way to test and keep alive secure communication links without the need to renegotiate the connection each time. In 2011, one of the RFC's authors, Robin Seggelmann, then a Ph.D. student at the Fachhochschule Münster, implemented the Heartbeat Extension for OpenSSL. Following Seggelmann's request to put the result of his work into OpenSSL, his change was reviewed by Stephen N. Henson, one of OpenSSL's four core developers. Henson failed to notice a bug in Seggelmann's implementation, and introduced the flawed code into OpenSSL's source code repository on December 31, 2011. The flaw spread with the release of OpenSSL version 1.0.1 on March 14, 2012. Heartbeat support was enabled by default, causing affected versions to be vulnerable.

Discovery

According to Mark J. Cox of OpenSSL, Neel Mehta of Google's security team reported Heartbleed on April 1, 2014.

The bug was named by an engineer at Codenomicon, a Finnish cybersecurity company that also created the bleeding heart logo and launched the domain heartbleed.com to explain the bug to the public. According to Codenomicon, Google's security team reported Heartbleed to OpenSSL first, but both Google and Codenomicon discovered it independently. Codenomicon reports April 3, 2014 as their date of discovery and their date of notification of NCSC-FI for vulnerability coordination.

The Sydney Morning Herald published a timeline of the discovery on April 15, 2014, showing that some organizations had been able to patch the bug before its public disclosure. In some cases, it is not clear how they found out.

Bugfix and deployment

Bodo Moeller and Adam Langley of Google prepared the fix for Heartbleed. The resulting patch was added to Red Hat's issue tracker on March 21, 2014. Stephen N. Henson applied the fix to OpenSSL's version control system on 7 April. The first fixed version, 1.0.1g, was released on the same day. As of June 21, 2014, 309,197 public web servers remained vulnerable.

Certificate renewal and revocation

According to Netcraft, about 30,000 of the 500,000+ X.509 certificates which could have been compromised due to Heartbleed had been reissued by April 11, 2014, although fewer had been revoked.

By May 9, 2014, only 43% of affected web sites had reissued their security certificates. In addition, 7% of the reissued security certificates used the potentially compromised keys. "By reusing the same private key, a site that was affected by the Heartbleed bug still faces exactly the same risks as those that have not yet replaced their SSL certificates," Netcraft stated. eWeek said, Heartbleed is "likely to remain a risk for months, if not years, to come."

Exploitation

The Canada Revenue Agency reported a theft of Social Insurance Numbers belonging to 900 taxpayers, and said that they were accessed through an exploit of the bug during a 6-hour period on April 8, 2014. After the discovery of the attack, the agency shut down its website and extended the taxpayer filing deadline from April 30 to May 5. The agency said it would provide anyone affected with credit protection services at no cost. On April 16, the RCMP announced they had charged an engineering student in relation to the theft with "unauthorized use of a computer" and "mischief in relation to data".

The UK parenting site Mumsnet had several user accounts hijacked, and its CEO was impersonated. The site later published an explanation of the incident saying it was due to Heartbleed and the technical staff patched it promptly.

Anti-malware researchers also exploited Heartbleed to their own advantage in order to access secret forums used by cybercriminals. Studies were also conducted by deliberately setting up vulnerable machines. For example, on April 12, 2014, at least two independent researchers were able to steal private keys from an experimental server intentionally set up for that purpose by CloudFlare. Also, on April 15, 2014, J. Alex Halderman, a professor at University of Michigan, reported that his honeypot server, an intentionally vulnerable server designed to attract attacks in order to study them, had received numerous attacks originating from China. Halderman concluded that because it was a fairly obscure server, these attacks were probably sweeping attacks affecting large areas of the Internet.

In August 2014, it was made public that the Heartbleed vulnerability enabled hackers to steal security keys from Community Health Systems, the second-biggest for-profit U.S. hospital chain in the United States, compromising the confidentiality of 4.5 million patient records. The breach happened a week after Heartbleed was first made public.

Possible prior knowledge and exploitation

Many major web sites patched the bug or disabled the Heartbeat Extension within days of its announcement, but it is unclear whether potential attackers were aware of it earlier and to what extent it was exploited.

Based on examinations of audit logs by researchers, it has been reported that some attackers may have exploited the flaw for at least five months before discovery and announcement. Errata Security pointed out that a widely used non-malicious program called "Masscan", introduced six months before Heartbleed's disclosure, abruptly terminates the connection in the middle of handshaking in the same way as Heartbleed, generating the same server log messages, adding "Two new things producing the same error messages might seem like the two are correlated, but of course, they aren't".

According to Bloomberg News, two unnamed insider sources informed it that the United States' National Security Agency had been aware of the flaw since shortly after its appearance but—​​instead of reporting it—​​kept it secret among other unreported zero-day vulnerabilities in order to exploit it for the NSA's own purposes. The NSA has denied this claim, as has Richard A. Clarke, a member of the National Intelligence Review Group on Intelligence and Communications Technologies that reviewed the United States' electronic surveillance policy; he told Reuters on April 11, 2014 that the NSA had not known of Heartbleed. The allegation prompted the American government to make, for the first time, a public statement on its zero-day vulnerabilities policy, accepting the recommendation of the review group's 2013 report that had asserted "in almost all instances, for widely used code, it is in the national interest to eliminate software vulnerabilities rather than to use them for US intelligence collection", and saying that the decision to withhold should move from the NSA to the White House.

Behavior

A depiction of Heartbleed

The RFC 6520 Heartbeat Extension tests TLS/DTLS secure communication links by allowing a computer at one end of a connection to send a "Heartbeat Request" message, consisting of a payload, typically a text string, along with the payload's length as a 16-bit integer. The receiving computer then must send exactly the same payload back to the sender.

The affected versions of OpenSSL allocate a memory buffer for the message to be returned based on the length field in the requesting message, without regard to the actual size of that message's payload. Because of this failure to do proper bounds checking, the message returned consists of the payload, possibly followed by whatever else happened to be in the allocated memory buffer.

Heartbleed is therefore exploited by sending a malformed heartbeat request with a small payload and large length field to the vulnerable party (usually a server) in order to elicit the victim's response, permitting attackers to read up to 64 kilobytes of the victim's memory that was likely to have been used previously by OpenSSL. Where a Heartbeat Request might ask a party to "send back the four-letter word 'bird'", resulting in a response of "bird", a "Heartbleed Request" (a malicious heartbeat request) of "send back the 500-letter word 'bird'" would cause the victim to return "bird" followed by whatever 496 characters the victim happened to have in active memory. Attackers in this way could receive sensitive data, compromising the confidentiality of the victim's communications. Although an attacker has some control over the disclosed memory block's size, it has no control over its location, and therefore cannot choose what content is revealed.

Affected OpenSSL installations

The affected versions of OpenSSL are OpenSSL 1.0.1 through 1.0.1f (inclusive). Subsequent versions (1.0.1g and later) and previous versions (1.0.0 branch and older) are not vulnerable. Installations of the affected versions are vulnerable unless OpenSSL was compiled with -DOPENSSL_NO_HEARTBEATS.

Vulnerable program and function

The vulnerable program source files are t1_lib.c and d1_both.c and the vulnerable functions are tls1_process_heartbeat() and dtls1_process_heartbeat().

Patch

The problem can be fixed by ignoring Heartbeat Request messages that ask for more data than their payload needs.

Version 1.0.1g of OpenSSL adds some bounds checks to prevent the buffer over-read. For example, the following test has been added, which discards the Heartbeat Request preventing a reply from being constructed if it would trigger the Heartbleed:

if (1 + 2 + payload + 16 > s->s3->rrec.length) return 0; /* silently discard per RFC 6520 sec. 4 */

The version control system contains a complete list of changes.

Impact

The data obtained by a Heartbleed attack may include unencrypted exchanges between TLS parties likely to be confidential, including any form post data in users' requests. Moreover, the confidential data exposed could include authentication secrets such as session cookies and passwords, which might allow attackers to impersonate a user of the service.

An attack may also reveal private keys of compromised parties, which would enable attackers to decrypt communications (future or past stored traffic captured via passive eavesdropping, unless perfect forward secrecy is used, in which case only future traffic can be decrypted if intercepted via man-in-the-middle attacks).

An attacker having gained authentication material may impersonate the material's owner after the victim has patched Heartbleed, as long as the material is accepted (for example, until the password is changed or the private key revoked). Heartbleed therefore constitutes a critical threat to confidentiality. However, an attacker impersonating a victim may also alter data. Indirectly, Heartbleed's consequences may thus go far beyond a confidentiality breach for many systems.

A survey of American adults conducted in April 2014 showed that 60 percent had heard about Heartbleed. Among those using the Internet, 39 percent had protected their online accounts, for example by changing passwords or canceling accounts; 29 percent believed their personal information was put at risk because of the Heartbleed bug; and 6 percent believed their personal information had been stolen.

Client-side vulnerability

Although the bug received more attention due to the threat it represents for servers, TLS clients using affected OpenSSL instances are also vulnerable. In what The Guardian therefore dubbed "reverse Heartbleed", malicious servers are able to exploit Heartbleed to read data from a vulnerable client's memory. Security researcher Steve Gibson said of Heartbleed that "it's not just a server-side vulnerability, it's also a client-side vulnerability because the server, or whomever you connect to, is as able to ask you for a heartbeat back as you are to ask them."

The stolen data could contain usernames and passwords. Reverse Heartbleed affected millions of application instances. Some of the vulnerable applications are listed in the "Software applications" section below.

Specific systems affected

Cisco Systems has identified 78 of its products as vulnerable, including IP phone systems and telepresence (video conferencing) systems.

Websites and other online services

An analysis posted on GitHub of the most visited websites on April 8, 2014 revealed vulnerabilities in sites including Yahoo!, Imgur, Stack Overflow, Slate, and DuckDuckGo. The following sites have services affected or made announcements recommending that users update passwords in response to the bug:

• Akamai Technologies
• Amazon Web Services
• Ars Technica
• Bitbucket
• BrandVerity
• Freenode
• GitHub
• IFTTT
• Internet Archive
• Mojang
• Mumsnet
• PeerJ
• Pinterest
• Prezi
• Reddit
• Something Awful
• SoundCloud
• SourceForge
• SparkFun
• Stripe
• Tumblr
• All Wikimedia Foundation wikis (including Wikipedia in all languages)
• Wunderlist

The Canadian federal government temporarily shut online services of the Canada Revenue Agency (CRA) and several government departments over Heartbleed bug security concerns.

Platform maintainers like the Wikimedia Foundation advised their users to change passwords.

The servers of LastPass were vulnerable, but due to additional encryption and forward secrecy, potential attacks were not able to exploit this bug. However, LastPass recommended that its users change passwords for vulnerable websites.

The Tor Project recommended that Tor relay operators and hidden service operators revoke and generate fresh keys after patching OpenSSL, but noted that Tor relays use two sets of keys and that Tor's multi-hop design minimizes the impact of exploiting a single relay. 586 relays later found to be susceptible to the Heartbleed bug were taken off-line as a precautionary measure.

Game-related services including Steam, Minecraft, Wargaming.net, League of Legends, GOG.com, Origin, Sony Online Entertainment, Humble Bundle, and Path of Exile were affected and subsequently fixed.

Software applications

Vulnerable software applications include:

• Several Hewlett-Packard server applications, such as HP System Management Homepage (SMH) for Linux and Windows.
• Some versions of FileMaker 13
• LibreOffice 4.2.0 to 4.2.2 (fixed in 4.2.3)
• LogMeIn claimed to have "updated many products and parts of our services that rely on OpenSSL".
• Multiple McAfee products, in particular some versions of software providing anti-viral coverage for Microsoft Exchange, software firewalls, and McAfee Email and Web Gateways
• MySQL Workbench 6.1.4 and earlier
• Oracle MySQL Connector/C 6.1.0-6.1.3 and Connector/ODBC 5.1.13, 5.2.5-5.2.6, 5.3.2
• Oracle Big Data Appliance (includes Oracle Linux 6)
• Primavera P6 Professional Project Management (includes Primavera P6 Enterprise Project Portfolio Management)
• WinSCP (FTP client for Windows) 5.5.2 and some earlier versions (only vulnerable with FTP over TLS/SSL, fixed in 5.5.3)
• Multiple VMware products, including VMware ESXi 5.5, VMware Player 6.0, VMware Workstation 10 and the series of Horizon products, emulators and cloud computing suites

Several other Oracle Corporation applications were affected.

Operating systems/firmwares

Several GNU/Linux distributions were affected, including Debian (and derivatives such as Linux Mint and Ubuntu) and Red Hat Enterprise Linux (and derivatives such as CentOS, Oracle Linux 6 and Amazon Linux), as well as the following operating systems and firmware implementations:

• Android 4.1.1, used in various portable devices. Chris Smith writes in Boy Genius Report that just this one version of Android is affected but that it is a popular version of Android (Chitika claim 4.1.1 is on 50 million devices; Google describe it as less than 10% of activated Android devices). Other Android versions are not vulnerable as they either have heartbeats disabled or use an unaffected version of OpenSSL.
• Firmware for some AirPort base stations
• Firmware for some Cisco Systems routers
• Firmware for some Juniper Networks routers
• IPCop 2.1.3 and some earlier versions (fixed in 2.1.4)
• pfSense 2.1.0 and 2.1.1 (fixed in 2.1.2)
• Western Digital My Cloud product family firmware

Vulnerability testing services

Several services have been made available to test whether Heartbleed affects a given site. However, many services have been claimed to be ineffective for detecting the bug. The available tools include:

• Tripwire SecureScan
• AppCheck - static binary scan, from Codenomicon
• Arbor Network's Pravail Security Analytics
• Norton Safeweb Heartbleed Check Tool
• Heartbleed testing tool by a European IT security company
• Heartbleed Scanner by Italian cryptologist Filippo Valsorda
• Heartbleed Vulnerability Test by Cyberoam
• Critical Watch Free Online Heartbleed Tester
• Metasploit Heartbleed scanner module
• Heartbleed Server Scanner by Rehmann
• Lookout Mobile Security Heartbleed Detector, an app for Android devices that determines the OpenSSL version of the device and indicates whether the vulnerable heartbeat is enabled
• Heartbleed checker hosted by LastPass
• Online network range scanner for Heartbleed vulnerability by Pentest-Tools.com
• Official Red Hat offline scanner written in the Python language
• Qualys SSL Labs' SSL Server Test which not only looks for the Heartbleed bug, but can also find other SSL/TLS implementation errors.
• Browser extensions, such as Chromebleed and FoxBleed
• SSL Diagnos
• CrowdStrike Heartbleed Scanner - Scans routers, printers and other devices connected inside a network including intranet web sites.
• Netcraft Site Report - indicates whether a website's confidentiality could be jeopardized due to a past exploitation of Heartbleed by checking data from Netcraft's SSL Survey to determine whether a site offered the heartbeat TLS Extension prior to the Heartbleed disclosure. The Netcraft Extensions for Chrome, Firefox and Opera also perform this check, whilst looking for potentially compromised certificates.

Other security tools have added support for finding this bug. For example, Tenable Network Security wrote a plugin for its Nessus vulnerability scanner that can scan for this fault. The Nmap security scanner includes a Heartbleed detection script from version 6.45.

Sourcefire has released Snort rules to detect Heartbleed attack traffic and possible Heartbleed response traffic. Open source packet analysis software such as Wireshark and tcpdump can identify Heartbleed packets using specific BPF packet filters that can be used on stored packet captures or live traffic.

Remediation

Although installing fixed software (dynamic versions of the OpenSSL library and binaries using OpenSSL statically) is a necessary step to address Heartbleed, more work may be needed. Running applications depending on OpenSSL may continue to use in-memory vulnerable OpenSSL code until they have been restarted before they stop being vulnerable.

Furthermore, a system attacked via Heartbleed may have its confidentiality and even its integrity at risk even after the vulnerability itself has been fixed. To regain confidentiality and authenticity, compromised servers must regenerate all compromised private key-public key pairs and must revoke and replace all certificates linked to these. In general, all compromised authentication material (such as passwords) must be replaced since it is rarely possible to confirm that a system which was affected has not been compromised.

Consequences of Heartbleed may remain even after the vulnerability was fixed and all compromised authentication material revoked. Much more system-specific work may be needed to confirm or restore the system's integrity, which may have been compromised if attackers used compromised authentication material to impersonate users in order to perform unauthorized modifications.

Browser security certificate revocation awareness

Since Heartbleed threatened the privacy of private keys, users of a website which was compromised could suffer from Heartbleed's effects even after the website was fixed and its previous certificate revoked, if their browser is unaware of certification revocations. The Security Certificate Revocation Awareness Test does not test whether a system has the Heartbleed bug, but allows users to test whether their web browser lets them visit a website which uses a revoked security certificate.

Root causes, possible lessons, and reactions

Although evaluating the total cost of Heartbleed is difficult, eWEEK estimated US$500 million as a starting point.

David A. Wheeler's paper How to Prevent the next Heartbleed analyzes why Heartbleed wasn't discovered earlier, and suggests several techniques which could have led to a faster identification, as well as techniques which could have reduced its impact. According to Wheeler, the most efficient technique which could have prevented Heartbleed is an atypical test suite thoroughly performing what he calls "negative testing", i.e. testing that invalid inputs cause failures rather than successes. Wheeler highlights that a single general-purpose test suite could serve as a base for all TLS implementations.

According to an article on The Conversation written by Robert Merkel, Heartbleed revealed a "massive failure of risk analysis". Merkel thinks OpenSSL gives more importance to performance than to security, which no longer makes sense in his opinion. But Merkel considers that OpenSSL should not be blamed as much as OpenSSL users, who chose to use OpenSSL, without funding better auditing and testing. Merkel explains that 2 aspects determine the risk that more similar bugs will cause vulnerabilities. The library's source code influences the risk of writing bugs with such an impact. Secondly, OpenSSL's processes affect the chances of catching bugs quickly. On the first aspect, Merkel mentions the use of the C programming language as one risk factor which favored Heartbleed's appearance, echoing Wheeler's analysis.

On the same aspect, Theo de Raadt, founder and leader of the OpenBSD and OpenSSH projects, has criticized the OpenSSL developers for writing their own memory management routines and thereby, he claims, circumventing OpenBSD C standard library exploit countermeasures, saying "OpenSSL is not developed by a responsible team." Following Heartbleed's disclosure, members of the OpenBSD project forked OpenSSL into LibreSSL.

The author of the change which introduced Heartbleed, Robin Seggelmann, stated that he "missed validating a variable containing a length" and denied any intention to submit a flawed implementation. Following Heartbleed's disclosure, Seggelmann suggested focusing on the second aspect, stating that OpenSSL is not reviewed by enough people. Although Seggelmann's work was reviewed by an OpenSSL core developer, the review was also intended to verify functional improvements, a situation making vulnerabilities much easier to miss.

OpenSSL core developer Ben Laurie claimed that a security audit of OpenSSL would have caught Heartbleed. Software engineer John Walsh commented "Think about it, OpenSSL only has two [fulltime] people to write, maintain, test, and review 500,000 lines of business critical code." The OpenSSL foundation's president, Steve Marquess, said "The mystery is not that a few overworked volunteers missed this bug; the mystery is why it hasn't happened more often." David A. Wheeler described audits as an excellent way to find vulnerabilities in typical cases, but noted that "OpenSSL uses unnecessarily complex structures, which makes it harder to both humans and machines to review." He wrote:

There should be a continuous effort to simplify the code, because otherwise just adding capabilities will slowly increase the software complexity. The code should be refactored over time to make it simple and clear, not just constantly add new features. The goal should be code that is “obviously right”, as opposed to code that is so complicated that “I can’t see any problems”.

LibreSSL made a big code cleanup, removing more than 90,000 lines of C code just in its first week.

According to security researcher Dan Kaminsky, Heartbleed is sign of an economic problem which needs to be fixed. Seeing the time taken to catch this simple error in a simple feature from a "critical" dependency, Kaminsky fears numerous future vulnerabilities if nothing is done. When Heartbleed was discovered, OpenSSL was maintained by a handful of volunteers, only one of whom worked full-time. Yearly donations to the OpenSSL project were about US$2,000. The Heartbleed website from Codenomicon advised money donations to the OpenSSL project. After learning about donations for the 2 or 3 days following Heartbleed's disclosure totaling US$841, Kaminsky commented "We are building the most important technologies for the global economy on shockingly underfunded infrastructure." Core developer Ben Laurie has qualified the project as "completely unfunded". Although the OpenSSL Software Foundation has no bug bounty program, the Internet Bug Bounty initiative awarded US$15,000 to Google's Neel Mehta, who discovered Heartbleed, for his responsible disclosure.

Paul Chiusano suggested Heartbleed may have resulted from failed software economics.

The industry's collective response to the crisis was the Core Infrastructure Initiative, a multimillion-dollar project announced by the Linux Foundation on April 24, 2014 to provide funds to critical elements of the global information infrastructure. Founding backers include Amazon, Dell, Facebook, Fujitsu, Google, IBM, Intel, Microsoft, NetApp, Rackspace, VMware and The Linux Foundation. The initiative intends to allow lead developers to work full-time on their projects and to pay for security audits, hardware and software infrastructure, travel, and other expenses. OpenSSL is a candidate to become the first recipient of the initiative's funding.

External links

• Summary and Q&A about the bug by Codenomicon Ltd
• Information for Canadian organizations and individuals