Buffer Overflow attack encountered in one of the CTFs

NOTE: This blog should definitely not be treated as a way to learn reverse engineering. If you really want to learn reverse engineering, start with learning the low-level languages and assembly language programming. Then, dig deeper with x86 family architecture and their instruction set. An understanding of the digital logic circuits would also be helpful. There is no technique without these which will make you understand reverse engineering. Write your own simple programs and then reverse-engineer them for practice.

Last year, there were three questions in HackCon Main Event (now Build The Shield) which revolved around Buffer Overflow. Most of the students asked the solution to these questions as they found it interesting, but couldn’t completely solve it. So, here we start with one of the riddles which was present in the event.

An application called baa.exe was running on all the servers which said – “DO NOT CLOSE THIS!… blah blah” on the console. And if I remember it correctly, the binary was available in a folder called “ctfartifacts” in the C drive of the servers. The idea was to debug this executable and find the password. What follows is an account of one of the approaches that you could have tried with OllyDbg.

Prerequisites –

  • I assume that you understand how to program in C/C++.
  • I assume that you understand Assembly Language
  • I assume you have a basic understanding of registers and understand how stack works and how it grows in memory.
  • I assume that you are comfortable with commands like MOV, PUSH, LEA and yada yada yada.

A few things before we begin (a crisp short start which you should remember going below) –

  • Register ESP (Stack Pointer) – A 32-bit register which always points to the last element used on the stack. It’s implicitly manipulated by the CPU instructions.
  • Register EBP (Base Pointer) – A 32-bit register to reference all function parameters and function variables in the current stack frame.These can be manipulated only explicitly.
  • If you find “EBP + some number” (EBP+8 and onwards, like EBP + 8, EBP + 12, EBP + 16) in the instruction set, it means that the value being referred is a function argument.
  • If you find “EBP – some number”  in the instruction set, it means that the value being referred is a local variable to that function.
  • If an instruction set starts like


    we call it a function prolog. This is how every function starts. But keep in mind that if optimization is enabled, then all the accesses would be done through ESP itself and there wouldn’t be any PUSH/MOV. This is called frame pointer optimization.

  • If there is a function call, the EAX register generally stores its returned value.
  • Words are stored in memory in little-endian byte order for x86 architectures.

OllyDbg shortcuts –

  • F2 on any given line enables a breakpoint.
  • F9 runs the attached program until a debugging point is hit (if there is one).

Before we begin, ensure that there is a file called flag1.txt in this location – “C:\flags\flag1\flag1.txt” and write something in it which you want to treat as a flag. Now, let’s just start straight with the debugging. Start the exe and the above quoted message will appear on the prompt. If you run “netstat -A” from the command prompt, you will see a list of local addresses with ports on which an application is running. If you kill the above application and re-run the command, you would find that a line containing this IP address – “” and says “LISTENING” disappears (which reappears if you restart the program). This is kind of a giveaway that the application is running on port – #37517 and is waiting for a connection on this port.

Now fire up the OllyDbg and attach this running process (Select the process whose path says the path to your executing binary) to it. Let’s get to the execution module now.

View Modules

By default the binary loads in ntdll module. Almost all the modules available above are kernel modules, except for the very first SS module. Let’s select that module and then select Search for – “All referenced text strings” as below.

Referenced strings

Here you can see a list of all the hard coded strings in the binary. Now before we move ahead, let’s try connecting to the socket where this application is currently listening. You can directly telnet to it with port as 37517, if telnet client is installed on your machine or you can keep a script handy which you can use to connect to any socket connection. You can find one such script here.

If you run the above program, you would find that the socket sends a string “Please enter your password:” on connection and waits for receive. The string itself says that you need a password to crack this binary. Now, if I move out of reverser’s mindset and think of it as a developer, I would definitely do a string comparison just after I receive my required string. With this in mind, I select “Please enter your password: “ as my next destination in select referenced text string (a double click will land you on the line containing that string).

This is where I land if I select the string.

Enter passoword

If you look closely, you would realise that most of the red text use Ws2_32 library. Any application that uses WinSock must be linked with Ws2_32 library file. These are thus core function calls to WinSock library. If we observe the function call pattern, we find that just after the above hard-coded string, there is a send function call which takes 4 parameters. After send, there is a call to WSAGetLastError with no parameters, a call to printf (MSVCR library), a call to recv with 4 parameters again, then presumably two user-defined functions with 1 and 3 parameters respectively and then there is call to shutdown and closesocket with 2 & 1 parameters respectively. Except for the two user-defined function calls, all the calls refer to WinSock library (obviously except printf). Moreover, the user-defined function call happens post the recv call and just before the shutdown. Then, in all likelihood, all comparison with the password which the client socket sends would either happen above the user-defined function calls or in the user-defined functions themselves.

Now, let’s have a look at the instructions between those calls. The very first line after recv has a mov instruction where EAX (probably a function return value which makes sense as a recv call would return the number of bytes received) is being moved to a local variable. Post this instruction, we see two comparisons between this returned value & 0 and the returned value & 200 respectively. The instructions –


translate into –

if (returnValue <= 0) then Jump SomeAddressX
if (returnValue >= 0x200) then Jump SomeAddressX

We can merge the above two statements (as the address to jump to is same for both comparisons) as

if (returnValue <= 0 && returnValue >= 0x200) then Jump SomeAddressX

With this, we can make a safe assumption that the returned string’s length should be less than 0x200 (512 in decimal) bytes as length less than 0 for a string doesn’t make any sense and as the same jump statement becomes valid for length greater than 512 bytes, so length is definitely less than 512 bytes.

The next line after the second jump is the first interesting part of the challenge. We now have a comparison between a new local variable and a hex – 5446534D which translates into TFSM in little endian, i.e., the string – MSFT. Post this comparison, there is again a non-zero check and a jump statement appears which in a nutshell is something like this for clarity’s sake –

if (valueAtSomeAddress != “MSFT”) then Jump SomeAddressY

If the comparison is successful, however, a call to a user-defined function happens. Let’s examine this function and check how it looks like.


This again makes it obvious. The hard-coded string is clearly a path which which exists on your system and contains the flag for this hack. Thus, the comparison does appear like one being done with the key/password (Microsoft competition? MSFT? maybe?). But this password obviously is not going to be as easy to exploit. I mean how naive it would be to keep a password like MSFT in a Microsoft competition? In fact, if you try submitting “MSFT”, you won’t get the flag.

Now, let’s take a look at the function closely once again. Our cross-check begins from the recv function call as that is when the client input comes in. We have 4 parameters for recv call in WinSock and the 1st parameter takes the socket handle, 2nd takes the buffer (the client data), 3rd takes the length of the buffer and the 4th takes a set of flags that dictates how recv will behave. So, the second parameter is the client’s input point to the function. Even in case, you don’t know how recv works in Ws2_32, you’ll mark that there is only one LEA out of the 4 PUSH instructions involved in the function call.

CALL DWORD PTR DS:[<&WS2_32.#16>]

LEA stands for Load Effective Address. For any processing to happen on a value, you need to store it somewhere and there is only one local variable (EBP-38) which we can see that is being used for value storing. And to store any input, we already know that an address is required. So, LEA over (EBP-38) ensures that it is the variable which stores the user input.

Now, let’s take a step back and analyse. We have an entry point into the socket server with a maximum buffer length of 512 bytes. This value is being stored in some variable X. We know that this variable doesn’t have to do with the actual check however as the actual check happens with something else in the stack. But, the next interesting part is – the length of available memory available for first variable spills over the memory of the second variable. So, if I pass a long enough string, I can write the value to the next variable which, otherwise, we wouldn’t have access to. This was it.

With this hunch in mind, let’s try this method of overwriting values. The value starts being written off from EBP-38 and I have to reach EBP-18 to write “MSFT” to that variable. The difference between these locations, thus, is

|(EBP-38) – (EBP-18)| = 20

So, I have to write 20 hex (32 in decimal) bytes with some junk and then write MSFT. I have modified the above socket code to pass 32 junk characters followed by MSFT to test this. And as you would also see. This gives you the output.


You are free to choose anything.

A Tester can be the best attacker

A defender has to think of thousand ways, an attacker has to think of just one.

First job, first experience and the new hires group I am part of, decides to have a Hackathon for the new hires. A Hackathon that is true to its literal sense. I said “Why not? Let’s do it.” I always wanted to be on the other side of the event though.

I thought to myself if I’m going to conduct a Hackathon, why not conduct one that is Microsoft wide. Truth dawned and I realized that I should stop kidding. New into the company and you want to test the mettle of the ones who have been in the system for this long, that’s not just bold, that’s probably many levels ahead of that.

Voila! The bunch of folks who took this project actually wanted it to be at least Microsoft IT wide. The gang-leader still tried to keep it low profile until we have something substantial, but a Hackathon, which was MSIT wide was definitely on the cards. I was like a kid who has been told that you have been given the responsibility to test the mettle of the entire world.

We brainstormed a lot for hours and then, started the marathon coding sessions. I thought that we will do this and that and see how it pans out. But by the end of the day, it was pretty clear that I am still no one who can pull this event off. All I had was the lamest SQL hack that anyone can embed in an application and the simplest form of application that can be reverse engineered (which I later realized that I wasn’t supposed to code it anyway as that was part of another hacking event). And on that, another epic moment dawned on me, how are we going to identify that someone was hacked and who was that someone’s hacker.

Xkcd - 327

Bummer! All for so much enthusiasm. The gang-leader was still very much optimistic and ecstatic about whatever the day had delivered. I wanted to join him but all I could think of was either this guy is actually a pro or we are just destined to doom. We left at 2 that night. And to be honest, I had no clue what exactly did we do that took us so much time.

Taking a step back, one thing which was pretty clear in our mind was that there would be two different hacking events which would go on in parallel. The one that I was working on, was where every team would be given a server of their own which was running on a VM. A web form application, which had bugs, was deployed on these servers (VMs) with an extra catch that everything including the Application Pools were stopped. The other event consisted of a hackable central server. The latter event was more like a game where difficulty level increases with every difficulty one just surpasses.

Then began our next night-out-coding session. There were a few informal hangouts about how to go about completing this project, but this probably was the only proper coding session after the first night. There was something different about this session though. The moment I entered the conference room where we all were meeting, I was greeted with this giant and beautiful architecture of the whole system. God, that was big! The first thing that popped in my mind was who can actually even imagine drawing that. No perks for guessing, it was the gang-leader. He went on to explain how he envisions the whole architecture. The architecture wasn’t complicated per se. In fact, if I were to draw one such, I would also have drawn one which was something closer in resemblance. But the point is – It was just big… insanely big.

that big

Anyways, we started to code. I picked my designing part of the client side server. In our informal hangouts, we had discussed a few vulnerabilities that we can use and expose for exploitation. Still unclear about how to identify the hacker and who hacked whom, I picked on six or so vulnerabilities and started working on them.

A hacked webpage would lead the server script to crash. In other words, an exception would be thrown. It was this Error 500 that we relied on to identify the hack. A custom error page for Error 500 was designed. When a page crashed, it invoked this page. This error page, in turn looked for a shared folder where flags (GUIDs, basically) mapped to the corresponding user’s crashed page were stored. This GUID was displayed to the attacker which he had to submit to us so that we can identify the attacker and the one who was attacked. The folder was refreshed with newer flags to maintain extra cautiousness against a “friendly-play” (Friends generally follow the “tu bhai hai naa!” (Come on man! You’re my bro.) strategy. They’ll ask their losing friends to share their flags. The already losing friend thus decides that they are losing anyways, why not let your best friend win.)


Though this flag generation and distribution module might appear small, it was the most critical module in the whole architecture as it was the only way we could identify someone who did the attack and the guy who was attacked. To keep it totally aloof from any crash, it was again divided into two separate modules – one module just cared about the flag generation and the other one just cared about the distribution process. This helped in avoiding deadlock conditions and conditions where a lock could be on hold for long when the files were queued for multiple read operations. Any key assigned that was older than the currently generated one by a level of 2 became invalid. Further these were developed as Windows services which started on boot to keep it hidden from users’ view.

After 3-4 such marathons, we were ready with everything. Everything… Sigh! So much for the hullabaloo. Then the testing season started. And the initial test phases passed off with flying colors. I was actually ecstatic. We syspreped the VHDs for client and server thinking that everything that was supposed to be done is done and we were ready for the word Go! But, thanks for one curious fella who still wasn’t satisfied.

We sat one night together to test it out from head to toe in totally isolated conditions with none of our credentials involved. And then every horror that we could have imagined came to life. The feeling could be described in two simple words – Nothing worked.


The first thing that we found was, while creating the application, I left my credentials involved somewhere in it. So, when the system started in total isolation (as an administrator), the application looked for my credentials (which never existed in the first place on the server) instead of kicking off things as an administrator. Trying to debug it piece-by-piece became a pain in the butt. Realizing that it probably would become much more tedious, I created everything from scratch without using my credentials.

The next thing we found was, no matter how current the flags were and howsoever valid they happen to be, the central server said the flag being submitted by the capturer is invalid. We banged our heads to figure out the issue for nearly more than 3 hours but to no avail. With no plausible visible issue and out of frustration, I started counting the number of characters in the GUIDs (on someone’s suggestion, can’t recall who). Much Ado About Nothing. It turns out that flags being generated and the ones being distributed differed by that last always eluding character. But hey, anything that fixes your pending bug is soothing.

Then it turns out that a person can hack himself. That probably could have been the worst thing to have happened. All everyone had to do was submit all his flags when he failed in his endeavours to hack others. Yes, he would lose flags for submitting his own flag, but then he would get the points for at least submitting the flags, won’t he?

This whole thing was already blowing up in our face. What if something like this happens when the event is actually on? I can’t even fathom the consequences. Anyhow we held our senses instead of going into panic mode and carried on with our work.

blow up

This time to be extra cautious, we created multiple fake participants and started the game again. Surprise, surprise! It blew up again. The culprit this time again were the bloody flags. Though it may not appear as a big issue, it was very subtle. The distribution of the flags was pathetically slow. Be it a network issue or a processing power one, this should not happen. What if the flag expires before it is actually distributed by the distributing service. The flag owner will go on to be the best defender without even touching his system. Small fix, but it definitely needed one.

All it needs is that one last kick to make you feel that you just can’t do it. Feeling that somehow everything is working as it is supposed to work, I was beginning to feel now that maybe, just maybe, we can pull this off. But there has to be that one last thing. The last night before the event, we again tested the whole scenario from end to end. The issue this time couldn’t have been subtler. We realized that after about half an hour or so, everything just about came to a freeze. Something was eating up the whole memory. We always felt this but we never paid any heed to it, given the kind of issues that kept coming up. Even after a reboot, the same thing happened. Looking up the memory usage, it was clear that the utilization shot up just after a few minutes. It was just that it was almost a complete freeze which happened after that long duration. The issue this time was with an exploit we wanted the gamers to explore. But as the general bugs are, the developers don’t have any clue about it.

I, for one, actually felt as if I would get lost into the code just by looking at it. The code was just perfect, at least it appeared to be. Think man, you just don’t have that spirit. Which coding principles did you evade while coding which led to this? What could have been that blunder?!

Will work

Logs, they always come in handy! It is one thing which differentiates between a good and an awesome programmer. One doesn’t understand the essence of logs unless he experiences it firsthand. Always leave a trail somewhere so that you can think through your mess. The trail is your guide to improvement. When staring at the code didn’t work, we sorted to logging. A dummy log was created to look into the issue and there it was, right in our face! The issue this time was an open port which was left opened in the memory and was never closed. But ports? We didn’t do any port.open() thingy for that module. And here fellas, you realize another truth, “Why one should not rely on garbage collection!” There was a disposable object which was created and was supposed to be disposed which opened this port. So, we had to dispose it manually. Tired of all the staring, we syspreped for the upteempth time and then left with a sad face. Maybe this was just not supposed to be.

So what’s the point of this all? Why this big write up? Developers just write the code and think that they have won the battle. All it took was a few night outs to realize what a failure it can be if it were only the developers who drive this tech industry. Testers know their way and they know it well! They know how to break and what will make the system break. They are the dudes!

And as for the event, it went kickass! 🙂