How I Reported a DoS Vulnerability to AWS

As much as I love reading bug hunting stories, I enjoy writing them too. This story was waiting long in my drafts pending disclosure approval from two companies involved with this bug. In my bug hunting stories, I intend to document my thought process, which concentrates on what didn’t work as much as on what worked. So let’s get on with the story…

Chapter 1: Strange Behaviour Encountered
While hunting for bugs on a private program, I came across a weird endpoint. I had been hunting on this  platform for quite a few days until then. When I was going through the Sitemap on Burp, I realised an endpoint – ‘/administrator’. In so many days of my testing, I had never come across this endpoint. Obviously, this endpoint must have been hidden somewhere without an actual call to the endpoint ever being made and Burp picked it up through scraping. Excited on reading the word ‘administrator’, I jumped right into my browser and requested http://platform.com/administrator. In response, I got a JSON reading:

{"message": “[X.X.X.X] Thanks for the visit."}

where X.X.X.X was my IP address. “Hmm.. strange behaviour”, I thought at first. When I checked my Burp logs, I realized that this endpoint was redirecting to:

some-random-id.execute-api.eu-west-1.amazonaws.com/ProdStage/administrator

And this was the actual endpoint that was sending the above mentioned response. I tried to directory brute force it, but I got the same result on all requests. Not able to figure out what was going on, I decided to keep this aside and focus on finding bugs on the visible attack surface of the platform.

Chapter 2: It Got Stranger:
But guess what, I was no longer able to access the platform. Whenever I tried to, I was welcomed with CloudFront 403 Forbidden page:

Strange. Time for a hypothesis:

Hypothesis #1: My IP was blocked for visiting the /administrator endpoint

To test this hypothesis, I tried changing my IP address through a VPN and was able to access the platform again. Then I tried visiting this /administrator endpoint through the VPN. Now, when I tried accessing the site again – “Blocked”. So, Hypothesis #1 was proven. Visiting the /administrator gets your IP into some sort of a blacklist. My immediate intuition was that there is something very important at this endpoint for the platform to have it configured in such a restrictive way. So, I had to probe further.

Chapter 3: I thought I was on a winning trail
Excited to probe further, I built my next hypothesis.
Hypothesis #2: Something super critical is hidden behind this endpoint

I wanted to understand what was being used to perform build the blacklist. So, I did the following Google search:

execute-api "Thanks for the visit”

And the very first result was this SANS document: https://www.sans.org/cyber-security-summit/archives/file/summit-archive-1519149524.pdf.

On searching within the document for the term “Thanks for the visit”, I was thrown to some source code written on Page 27 of the document, an excerpt of which can be seen below.

On reading the heading, I was like: “WTF! A honeypot? I was lured by a honeypot and got myself blacklisted. Damn!”. So, Hypothesis #2 was disproven. There was nothing critical behind this endpoint.

Chapter 4: About To Drop Testing
I wasn’t finding anything interesting behind this endpoint and thought that I should drop testing it and figure out a way to get out of the blacklist by contacting the site support. As I was about to give up on this endpoint, I noticed this line of code in the SANS document:

source_ip = event['headers']['X-Forwarded-For'].encode('utf8').split(',')[0].strip()

“X-Forwarded-For, Interesting!”, I said to myself. So, the endpoint was using the XFF header to determine the IP address to be blacklisted. I had read bug reports earlier where the target server behaved differently when XFF header was sent by the client. Another hypothesis came to my mind.

Hypothesis #3: If I passed my own XFF header along with the request, I can fool the endpoint to pick the IP address in the header instead of picking my actual IP address.

Now, this was a far-fetched hypothesis which I had presumed to not work, but was anyways going to test it. So, lets see what happened when I tested this:

Well, well! My hypothesis was half proven. The endpoint responds with the IP address in the XFF header instead of returning my actual IP address. I say half-proven because I wanted to ensure that I am also able to get that IP address in the XFF header blocked. The test for this was simple.

Chapter 5: The Litmus Test

• Installed a VPN extension, called Betternet, for Chrome and enabled the VPN.
• Verified that I was able to access the site through the VPN.
• Checked my VPN public IP address by googling ‘What is my IP’.
• Sent the below request through Burp repeater where X.X.X.X was the VPN IP: (this request will not go through the VPN because the VPN is just running as an extension on Chrome)

GET /ProdStage/administrator HTTP/1.1
Host: some-random-id.execute-api.eu-west-1.amazonaws.com
X-Forwarded-For: X.X.X.X
Connection: close

• The response reflects back the IP address from the XFF header as expected. Now, it was time to check whether the VPN IP was actually blacklisted.
• I tried accessing the site through the same VPN and guess what! I was welcomed with a sweet CloudFront 403 Forbidden page. Hypothesis #3 proved!

Let me explain the impact of this vulnerability. An attacker who passes random IP addresses through the XFF header to this endpoint will end up blacklisting legitimate users from the platform. With enough brute force, the attacker can end up adding the entire public IP range to the blacklist effectively causing a massive Denial of Service. Also, an attacker can affect a platform’s Google ranking by blacklisting Google crawler IP addresses.

Chapter 6: Digging Further

Before reporting this right away to the program, my mind started to prepare another hypothesis. I asked myself, “Could this possibly be a problem with AWS?”.

Hypothesis #4: Since this endpoint is hosted on AWS, maybe this honeypot is part of an AWS service offering and a lot of people may be using it.

I tried to think of a Google Dork that could help me search for all such endpoints. But, then I realized that site admins would not be so stupid to have Google crawl this endpoint and block it from indexing the entire site. To see if I was right I headed to robots.txt of the site I was testing and there the very first entry of the Disallow directive was: /administrator. So, I knew right away that Google Dorking would be futile.

To know more about this endpoint, I started my research by googling “Bad Bot Parser Function” and somehow ended up on this Github Repository:

https://github.com/awslabs/aws-waf-security-automations

Specifically, on this file:

https://github.com/awslabs/aws-waf-security-automations/blob/master/source/access-handler/access-handler.py

The code in this file looked almost similar to the one I found in the SANS deck Page 27 above. This repository was owned by “Amazon Web Services – Labs” and it was a verified account, so this proved my hypothesis #4. This endpoint was offered by AWS as a feature for blocking automated bots, spiders from scraping the website.

Also, on reading the code I realized that this is an AWS Lambda function. Actually, I should have concluded this on seeing the “execute-api” endpoint itself. But, we learn from experience. If I had to report this to AWS, I should have concrete evidence that my hypothesis stands correct for this code as well and is not isolated to the program I was testing. Again, this can easily be proven by just looking at the handler function code. But, the auditor inside me said: “We need to find blood on the floor” to make a strong report.

Chapter 7: Demonstrating our hypothesis

Well, I followed the instructions on the repo to setup my own AWS Honeypot. This AWS CloudFormation Template made it trivial for me to set it up within 15 minutes. Once, I had this setup, and got the unique honeypot URL, it was time to test it out. I followed the same steps that I did for testing the program’s /administrator endpoint and saw that my Lambda function reflected back the IP address passed in the XFF header. So, this proved Hypothesis #4 yet again for a different sample.

Before, reporting this bug to AWS, I wanted to get to the root cause of why this was happening. I headed over to AWS CloudWatch for checking my Lambda instance logs. To give you some background, Lambda services expose an ‘event’ JSON object which can be used in the handler code. As can be seen from the access-handler.py source code, the source_ip is extracted as follows:

source_ip = event['headers']['X-Forwarded-For'].split(',')[0].strip()

So, I had to check what was the value of event[‘headers’][‘X-Forwarded-For’] in my logs. Below is a screenshot of one such log:

In the screenshot, 15.76.50.0 was the IP I passed through the XFF header and the masked IP was my actual IP.

I concluded from the logs that a reverse proxy was handling the request first and then setting the XFF header before forwarding it to our Lambda function. But instead of replacing the XFF header that was passed by the client, it appended IP addresses to the request. So, the user passed XFF header IP address will always end up in the first position in the comma separated list of IP addresses. So, when the above line of code splits the value by commas and picks the 0th element of the resultant array, it will always end up with the user-passed XFF header IP address.

Chapter 8: The Report

I reported this to AWS and also to the H1 program through which I found this. The engineers at the program company were quick to fix it and they also pointed me to the repo which holds the vulnerable code. They suggested that I open a public GitHub issue to have the repo fix it. But since this had security implications, I decided against opening a public issue at that point.

After exchanging a few emails with the AWS Security Team, they acknowledged the issue and suggested that I open up a pull request to get the attention of the repository owners. This got me a bit excited because I always wanted to contribute to open source projects and had never got a good use case where I could contribute.

Chapter 9: The Fix

I contributed one line of code to the repository to fix this vulnerability and I created my first ever GitHub pull request:

https://github.com/awslabs/aws-waf-security-automations/pull/123

Note the screenshot of the CloudWatch log above. The actual IP address of the client can be seen in two places:
1. event[‘headers’][‘X-Forwarded-For’]
2. event[‘requestContext’][‘identity’][‘sourceIp’]

For event[‘headers’][‘X-Forwarded-For’], I could see that the actual IP address is always the second to last element in the comma-separated list of IP addresses in the XFF header. So, instead of picking 0th element from the comma separated array, the code could pick (len-2)nd element. That would ensure that the correct IP address always gets picked. However, this solution is not future-proof. Say, a change is made to the network architecture which adds or removes components between the reverse proxy server and the endpoint and that new component is now adding its own IP address to the list. This would cause the reverse proxy server’s IP to be the 2nd last element and the endpoint would again end up blacklisting the wrong IP address.

event[‘requestContext’][‘identity’][‘sourceIp’], however, was quite reliable and free of any ambiguity. The actual IP address is just a string which can simply be sent to the WAF for blacklisting, avoiding the unnecessary logic of splitting and picking the source IP address. Here’s the simple code diff for my fix:

-        source_ip = event['headers']['X-Forwarded-For'].split(',')[0].strip()
+        source_ip = event['requestContext']['identity']['sourceIp']

This bug hunting experience was a great learning experience for me. It was fun as it allowed me to learn a lot of new things and really made me think from the defensive side as well.

Credits

Kudos to Dan and Zack from AWS Security for being supportive throughout the process.

Something about the format of my bounty hunting stories:

Thank you for reading so far. When trying to read bug hunting articles, I always try to understand the thought process of the hacker behind the find. Just concentrating on the PoC won’t take us far. Understanding and adapting your thought process is key to bounty hunting. Hence, instead of just writing what worked, I focus on writing what did not work on way to the final step that led to the attack.

Happy Hacking!