Buffer Overflow attack encountered in one of the CTFs

NOTE: This blog should definitely not be treated as a way to learn reverse engineering. If you really want to learn reverse engineering, start with learning the low-level languages and assembly language programming. Then, dig deeper with x86 family architecture and their instruction set. An understanding of the digital logic circuits would also be helpful. There is no technique without these which will make you understand reverse engineering. Write your own simple programs and then reverse-engineer them for practice.

Last year, there were three questions in HackCon Main Event (now Build The Shield) which revolved around Buffer Overflow. Most of the students asked the solution to these questions as they found it interesting, but couldn’t completely solve it. So, here we start with one of the riddles which was present in the event.

An application called baa.exe was running on all the servers which said – “DO NOT CLOSE THIS!… blah blah” on the console. And if I remember it correctly, the binary was available in a folder called “ctfartifacts” in the C drive of the servers. The idea was to debug this executable and find the password. What follows is an account of one of the approaches that you could have tried with OllyDbg.

Prerequisites –

  • I assume that you understand how to program in C/C++.
  • I assume that you understand Assembly Language
  • I assume you have a basic understanding of registers and understand how stack works and how it grows in memory.
  • I assume that you are comfortable with commands like MOV, PUSH, LEA and yada yada yada.

A few things before we begin (a crisp short start which you should remember going below) –

  • Register ESP (Stack Pointer) – A 32-bit register which always points to the last element used on the stack. It’s implicitly manipulated by the CPU instructions.
  • Register EBP (Base Pointer) – A 32-bit register to reference all function parameters and function variables in the current stack frame.These can be manipulated only explicitly.
  • If you find “EBP + some number” (EBP+8 and onwards, like EBP + 8, EBP + 12, EBP + 16) in the instruction set, it means that the value being referred is a function argument.
  • If you find “EBP – some number”  in the instruction set, it means that the value being referred is a local variable to that function.
  • If an instruction set starts like


    we call it a function prolog. This is how every function starts. But keep in mind that if optimization is enabled, then all the accesses would be done through ESP itself and there wouldn’t be any PUSH/MOV. This is called frame pointer optimization.

  • If there is a function call, the EAX register generally stores its returned value.
  • Words are stored in memory in little-endian byte order for x86 architectures.

OllyDbg shortcuts –

  • F2 on any given line enables a breakpoint.
  • F9 runs the attached program until a debugging point is hit (if there is one).

Before we begin, ensure that there is a file called flag1.txt in this location – “C:\flags\flag1\flag1.txt” and write something in it which you want to treat as a flag. Now, let’s just start straight with the debugging. Start the exe and the above quoted message will appear on the prompt. If you run “netstat -A” from the command prompt, you will see a list of local addresses with ports on which an application is running. If you kill the above application and re-run the command, you would find that a line containing this IP address – “” and says “LISTENING” disappears (which reappears if you restart the program). This is kind of a giveaway that the application is running on port – #37517 and is waiting for a connection on this port.

Now fire up the OllyDbg and attach this running process (Select the process whose path says the path to your executing binary) to it. Let’s get to the execution module now.

View Modules

By default the binary loads in ntdll module. Almost all the modules available above are kernel modules, except for the very first SS module. Let’s select that module and then select Search for – “All referenced text strings” as below.

Referenced strings

Here you can see a list of all the hard coded strings in the binary. Now before we move ahead, let’s try connecting to the socket where this application is currently listening. You can directly telnet to it with port as 37517, if telnet client is installed on your machine or you can keep a script handy which you can use to connect to any socket connection. You can find one such script here.

If you run the above program, you would find that the socket sends a string “Please enter your password:” on connection and waits for receive. The string itself says that you need a password to crack this binary. Now, if I move out of reverser’s mindset and think of it as a developer, I would definitely do a string comparison just after I receive my required string. With this in mind, I select “Please enter your password: “ as my next destination in select referenced text string (a double click will land you on the line containing that string).

This is where I land if I select the string.

Enter passoword

If you look closely, you would realise that most of the red text use Ws2_32 library. Any application that uses WinSock must be linked with Ws2_32 library file. These are thus core function calls to WinSock library. If we observe the function call pattern, we find that just after the above hard-coded string, there is a send function call which takes 4 parameters. After send, there is a call to WSAGetLastError with no parameters, a call to printf (MSVCR library), a call to recv with 4 parameters again, then presumably two user-defined functions with 1 and 3 parameters respectively and then there is call to shutdown and closesocket with 2 & 1 parameters respectively. Except for the two user-defined function calls, all the calls refer to WinSock library (obviously except printf). Moreover, the user-defined function call happens post the recv call and just before the shutdown. Then, in all likelihood, all comparison with the password which the client socket sends would either happen above the user-defined function calls or in the user-defined functions themselves.

Now, let’s have a look at the instructions between those calls. The very first line after recv has a mov instruction where EAX (probably a function return value which makes sense as a recv call would return the number of bytes received) is being moved to a local variable. Post this instruction, we see two comparisons between this returned value & 0 and the returned value & 200 respectively. The instructions –


translate into –

if (returnValue <= 0) then Jump SomeAddressX
if (returnValue >= 0x200) then Jump SomeAddressX

We can merge the above two statements (as the address to jump to is same for both comparisons) as

if (returnValue <= 0 && returnValue >= 0x200) then Jump SomeAddressX

With this, we can make a safe assumption that the returned string’s length should be less than 0x200 (512 in decimal) bytes as length less than 0 for a string doesn’t make any sense and as the same jump statement becomes valid for length greater than 512 bytes, so length is definitely less than 512 bytes.

The next line after the second jump is the first interesting part of the challenge. We now have a comparison between a new local variable and a hex – 5446534D which translates into TFSM in little endian, i.e., the string – MSFT. Post this comparison, there is again a non-zero check and a jump statement appears which in a nutshell is something like this for clarity’s sake –

if (valueAtSomeAddress != “MSFT”) then Jump SomeAddressY

If the comparison is successful, however, a call to a user-defined function happens. Let’s examine this function and check how it looks like.


This again makes it obvious. The hard-coded string is clearly a path which which exists on your system and contains the flag for this hack. Thus, the comparison does appear like one being done with the key/password (Microsoft competition? MSFT? maybe?). But this password obviously is not going to be as easy to exploit. I mean how naive it would be to keep a password like MSFT in a Microsoft competition? In fact, if you try submitting “MSFT”, you won’t get the flag.

Now, let’s take a look at the function closely once again. Our cross-check begins from the recv function call as that is when the client input comes in. We have 4 parameters for recv call in WinSock and the 1st parameter takes the socket handle, 2nd takes the buffer (the client data), 3rd takes the length of the buffer and the 4th takes a set of flags that dictates how recv will behave. So, the second parameter is the client’s input point to the function. Even in case, you don’t know how recv works in Ws2_32, you’ll mark that there is only one LEA out of the 4 PUSH instructions involved in the function call.

CALL DWORD PTR DS:[<&WS2_32.#16>]

LEA stands for Load Effective Address. For any processing to happen on a value, you need to store it somewhere and there is only one local variable (EBP-38) which we can see that is being used for value storing. And to store any input, we already know that an address is required. So, LEA over (EBP-38) ensures that it is the variable which stores the user input.

Now, let’s take a step back and analyse. We have an entry point into the socket server with a maximum buffer length of 512 bytes. This value is being stored in some variable X. We know that this variable doesn’t have to do with the actual check however as the actual check happens with something else in the stack. But, the next interesting part is – the length of available memory available for first variable spills over the memory of the second variable. So, if I pass a long enough string, I can write the value to the next variable which, otherwise, we wouldn’t have access to. This was it.

With this hunch in mind, let’s try this method of overwriting values. The value starts being written off from EBP-38 and I have to reach EBP-18 to write “MSFT” to that variable. The difference between these locations, thus, is

|(EBP-38) – (EBP-18)| = 20

So, I have to write 20 hex (32 in decimal) bytes with some junk and then write MSFT. I have modified the above socket code to pass 32 junk characters followed by MSFT to test this. And as you would also see. This gives you the output.


You are free to choose anything.