How to Protect Against OpenSSL 1.1.0a Vulnerability CVE-2016-6309

By on Dec 12, 2016

This blog post was written by Rock Liu.

Recently the OpenSSL security library gained a fix for a critical security issue (CVE-2016-6309) that affects OpenSSL Version 1.1.0a. The remote attackers can cause the OpenSSL server to crash, or execute arbitrary code on it, by simply sending a handshake packet with a message larger than 16KB. To defend against these attacks we analyzed the code, wrote a proof of concept, and prepared a signature for our customers.

OpenSSL offers the following description:

“The buffer to receive messages is initialized to 16KB. If a message is received that is larger than that, then the buffer is ‘realloc’d.’ This can cause the location of the underlying buffer to change. Anything that is referring to the old location will be referring to free’d data.”

From the OpenSSL source code, we know it uses the structure “ssl_st” to store the SSL session information. Both init_msg and init_buf are members of the structure of ssl_st. Init_msg points to the handshake message body, which is contained in the buffer pointed to by init_buf. The variable “s” (s->init_msg) is a pointer to the structure of ssl_st.

From this information, we suspect a single handshake message might trigger this vulnerability.

This is the format of a handshake packet in the TLS protocol:


| 0x16 | 0x0301 | 0x4000 | 0x0100 | 0x5560 | DATA(length=0x5560) |


|       |       |               |————–Length of challenge

|       |       |

|       |       |—————————————-Length handshake message

|       |—————————————————–Version

|——————————————————————Content type:Handshake


We manually constructed a handshake packet of more than 16KB (0x5560).


Figure 1: The TLS handshake packet.

We sent the oversized handshake to a vulnerable OpenSSL server, and it crashed.



Figure 2: Crash information from the vulnerable OpenSSL server.

Now for more details. First, let’s take a look at how the issue was fixed. The patch is available here.

if (!SSL_IS_DTLS(s)

&& s->s3->tmp.message_size > 0

–                    && !BUF_MEM_grow_clean(s->init_buf,

–                                           (int)s->s3->tmp.message_size

–                                           + SSL3_HM_HEADER_LENGTH)) {

+                    && !grow_init_buf(s, s->s3->tmp.message_size

+                                         + SSL3_HM_HEADER_LENGTH)) {

ssl3_send_alert(s, SSL3_AL_FATAL, SSL_AD_INTERNAL_ERROR);



The lines preceded by “-” are the old code, while the “+” lines are the patch. The patch introduced a new function, grow_init_buf, to replace the function BUF_MEM_grow_clean. This new function handles the chores of buffer reallocation to make sure the buffer is big enough to hold the handshake message.

Now let’s check out the new function: grow_init_buf:

static int grow_init_buf(SSL *s, size_t size) {

size_t msg_offset = (char *)s->init_msg – s->init_buf->data;

if (!BUF_MEM_grow_clean(s->init_buf, (int)size))

return 0;

if (size < msg_offset)

return 0;

s->init_msg = s->init_buf->data + msg_offset;   <——————reset the init_msg

return 1;


This newly introduced function internally will still call the old function BUF_MEM_grow_clean; after that it resets the init_msg (marked above).

Next, let’s look into this piece of vulnerable code with a debugger.


Figure 3: The unpatched code before the control flow reaches the function BUF_MEM_grow_clean.

Figure 3 gives us the following status information:

  • s->init_msg=0x6c3184
  • s->
  • length=0x4000 and is controllable by the attacker through the handshake message

Now s->init_msg points to our handshake message, and is contained in the buffer pointed to by s->


Figure 4: The scenario in which the control flow returns only from the function BUF_MEM_grow_clean.

At this point, apparently has been changed, but init_msg still points to 0x6c3184.

  • s->init_msg=0x6c3184
  • s->

Now let’s step into the function BUF_MEM_grow_clean to find out why s-> has changed:

size_t BUF_MEM_grow_clean(BUF_MEM *str, size_t len)


char *ret;

size_t n;

if (str->length >= len) {

if (str->data != NULL)

memset(&str->data[len], 0, str->length – len);

str->length = len;

return (len);


if (str->max >= len) {

memset(&str->data[str->length], 0, len – str->length);

str->length = len;

return (len);




return 0;


n = (len + 3) / 3 * 4;

if ((str->flags & BUF_MEM_FLAG_SECURE))

ret = sec_alloc_realloc(str, n);       <—–realloc a new memory for init_buf


ret = OPENSSL_clear_realloc(str->data, str->max, n);

if (ret == NULL) {


len = 0;

} else {

str->data = ret;

str->max = n;

memset(&str->data[str->length], 0, len – str->length);

str->length = len;


return (len);


In the preceding code, we have marked the place where memory reallocation occurs. If we can pass all three checks (“str->length >= len,” “str->max >= len,” and “len > LIMIT_BEFORE_EXPANSION”), the function would eventually change s-> (str->data) with the memory allocated by sec_alloc_realloc (marked).


Figure 5: The values of variables str->length, len, and str->max. All three variables can be controlled by the attacker.

From Figure 5, we can clearly see the values of variables when the program is executing inside the BUF_MEM_grow_clean:

  • len=0x5564 (the packet’s real length)
  • str->length=0x4000 (the length of the handshake message)
  • str->max=0x5558 (the length of challenge)

Because we can control the value of all three variables, it is trivial to bypass all the checks and hit the sec_alloc_realloc function.

Finally, let’s see what happens inside the function sec_alloc_realloc:

static char *sec_alloc_realloc(BUF_MEM *str, size_t len)


char *ret;

ret = OPENSSL_secure_malloc(len);          <———realloc memory

if (str->data != NULL) {

if (ret != NULL)

memcpy(ret, str->data, str->length);

OPENSSL_secure_free(str->data);       <———free old memory


return (ret);


The function sec_alloc_realloc reallocates new memory, and then frees the old memory (marked). At this point, however, s->init_msg is still pointing to the old memory that has been freed by sec_alloc_realloc. Thus later when s->init_msg is referenced, the program will either crash, or in some cases be forced to access the attacker-controlled memory, which may lead to code execution.

The patch added the new function grow_init_buf to replace the function BUF_MEM_grow_clean, and the new function will update init_msg when OpenSSL reallocates new memory to make sure it points to valid memory. For better visualization, we prepared a video to demonstrate the attack in action. (In the video, the right window shows an OpenSSL server on port 443 with standard parameters. In the left window, we send an oversized handshake to the vulnerable server, which crashes.)



More than 60% of active websites use OpenSSL to transfer data. More and more sites now use OpenSSL to protect the sensitive user information, such as passwords, card IDs, and usernames. Amazon, the world’s largest cloud vendor, recommend OpenSSL to its customers, and offers many advisories.

Many of us remember the notorious OpenSSL Heartbleed attack, which occurred in April 2014. At that time it was considered the Internet’s worst nightmare. “A survey of American adults conducted in April 2014 showed that 60% had heard about Heartbleed. Among those using the Internet, 39% had protected their online accounts, for example by changing passwords or canceling accounts; 29% believed their personal information was put at risk because of the Heartbleed bug; and 6% believed their personal information had been stolen,” according to Wikipedia. From the attack and these survey results, we can see that a single critical OpenSSL issue can cause a significant impact on the Internet due to OpenSSL’s wide deployment. The vulnerability, CVE-2016-6309, that we discuss in this post is different from other OpenSSL vulnerabilities. In this case, the flaw occurs when the OpenSSL server handles the handshake message, the first packet of the TLS protocol. This makes the attack easy to carry out because the attacker can send just one packet to deny access to the server without the need for authentication.

The update is available here.

For McAfee Network Security Platform customers, we have released the signature 0x45c08f00 “SSL: Possible OpenSSL Use-After-Free Vulnerability (CVE-2016-6309)” to prevent this attack.

About the Author

McAfee Labs

McAfee Labs is one of the leading sources for threat research, threat intelligence, and cybersecurity thought leadership. See our blog for more information.

Read more posts from McAfee Labs

Categories: McAfee Labs

Subscribe to McAfee Securing Tomorrow Blogs