Learning Reverse Engineering: A Step-By-Step Series | Part 01

Dive with me into the fundamentals of reverse engineering. Learn about x86 assembly, CPU registers, the instruction pipeline, essential terms and concepts for your journey into reverse engineering.
Learning Reverse Engineering: A Step-By-Step Series | Part 01

Hi folks! I hope 2026 has been nice to you so far and that you're doing well.

Soon, a training at work will take me into the world of reverse engineering, of which i have.. not have had a clue until very recently? haha. I am still a big noob on the topic, so I thought this the perfect timing to do this series, my first one into 2026 apparently.

Come learn Reverse Engineering basics with me! Even if you, or I, don't end up deeper down that path, it is still filled to the brim with important and fascinating topics learning about will always be worth it!

So, let's "learn out loud" here and cement more and more complex topics going down this specific rabbithole.

Have you heard of Reverse Engineering?

Reverse Engineering is one of the most critical disciplines in Software Security. It is the art of deconstructing a compiled binary to understand its logic, hidden features, or malicious intent.. essentially solving a complex puzzle without the original blueprint? Might sound rather stressfull haha, but we'll get there.

Famous Examples

... of software/ malware that was reverse-engineered to extraordinary results:

Stuxnet: Stuxnet was reverse-engineered to to discover it targeted specific Siemens PLCs in nuclear centrifuges, it's inner mechanisms and anit-analysis capabilities. [1]
Samba: RE of the SMB protocol to allow Linux to talk to Windows networks. [2]
Magic Lantern: RE of Canon firmware to add features to cameras. [3]


  1. X86byte. (o. D.). GitHub - x86byte/Stuxnet-Rootkit: Stuxnet extracted binaries by reversing & Stuxnet Rootkit Analysis. GitHub. https://github.com/x86byte/Stuxnet-Rootkit ↩︎

  2. How Samba was written | Hacker News. (o. D.). https://news.ycombinator.com/item?id=488089 ↩︎

  3. Wikipedia contributors. (2025, 25. September). Magic Lantern (firmware). Wikipedia. https://en.wikipedia.org/wiki/Magic_Lantern_(firmware) ↩︎

That's FASCINATING, isn't it? WE got these softwares, protocols, analyses.. through (very sophisticated) reverse engineering!

Enough fanboying, shall we start?

The Setup: Tools of the Trade

I really liked reading this article on tools to be used; as preparation for the training, I got IDAFree installed, I will try to tell you as much as i can while learning the tool myself!

Don't worry about setting up 5 tools reading this post, we'll get to it soon enough..

The Reverse Engineering Glossary: Essential Concepts

This glossary serves as a technical foundation for understanding how high-level logic is distilled into machine-readable execution.

Core Translation Terms

  • Compiler: A program that translates human-readable high-level code (like C) into machine-specific instructions. It performs syntax analysis, optimization, and finally generates a binary executable.
  • Bytecode / Machine Code: The lowest level of software, consisting of binary expressions (Opcodes) that the CPU executes directly. In a hex editor, this is viewed as pairs of hexadecimal digits.
00000000: 5468 6973 2069 7320 6120 7465 7374 2074  This is a test t
00000010: 6578 7466 696c 652e 0a                   extfile..
This, for example, is the bytecode of a testfile i created called test.txt. You can view this on your Linux machine with xxd test.txt, showing you the highlighted bytecode.
  • Assembly (Asm): A human-readable representation of machine code using mnemonics (like MOV, ADD). It is processor-specific; code for an x86 CPU will not run on an MSP430 (a widely used microcontroller).
I am going to use the MSP 430 as an example more often here, as it's structure is rather simple and you can find the architecture online in a beginner-friendly manner to read more about "mini-computers" like that for yourself, to better grasp complex topics where one shouldn't start with a full-on desktop pc.
  • Disassembly: The process of using a tool (a disassembler) to translate raw bytecode back into Assembly mnemonics. This is the primary window for reverse engineering.

Execution & Architecture

  • Offset / Address: A specific numerical location within the program’s memory space. In a 16-bit system, like that of the MSP430, addresses range from 0x0000 to 0xFFFF.
  • Hex-Code (Hexadecimal): A base-16 numbering system (0-9, A-F) used to represent binary data compactly. One byte (8 bits) is represented by two hex digits (e.g., 0xFF).
  • Register: A small, extremely fast storage location internal to the CPU used to hold data, addresses, or the status of the last operation.
  • ALU (Arithmetic Logic Unit): The "engine" of the CPU responsible for mathematical (addition, subtraction) and logical (AND, OR, XOR) operations.
  • Instruction Pointer (IP/PC): A special register that holds the memory address of the next instruction to be executed.
  • Subroutine: A set of instructions designed to perform a frequently used operation within a program. Think of it as a low-level (Assembly-level actually) version of a function that you can call and return to when needed for repetative tasks.

Data & Hardware Interaction

  • Word: The natural data size of a processor. For a 16-bit CPU, a word is 16 bits (2 bytes); for a 64-bit CPU, it is 64 bits (8 bytes).
  • Little-Endian: A memory architecture where the Least Significant Byte (LSB) is stored at the lowest address. If you view a 16-bit word 0x1234 in a byte-wise hex editor, it appears as 34 12. The opposite of that is called Big-Endian, storing the LSB at the highest address.
  • Port: A group of pins (usually 8) used for digital I/O. Ports allow the CPU to interact with external hardware like sensors or LEDs.
  • Memory-Mapped I/O: A design where hardware peripherals (like timers or ports) are assigned specific addresses in the normal memory map. The CPU interacts with hardware simply by reading or writing to these "special" memory addresses.
  • Stack: A "Last-In, First-Out" (LIFO) memory structure used to store local variables and return addresses during function calls. It typically grows downwards in memory.
We just introduced a very important term, the Stack, rather low-key. Let's take a minute and take a closer look at terminology and base concepts of memory, as that is vital to us learning more about binaries and reversing them.

Memory Layout: A Researcher’s Cheat Sheet

In computer science, we categorize memory into distinct segments. Understanding where data lives is the first step in identifying vulnerabilities like buffer overflows.

Segment Purpose Permissions Persistence
Code / Text Stores the executable machine instructions. Read / Execute Non-volatile (Stored in Flash/ROM)
Data Segment Stores initialized global and static variables. Read / Write Volatile (RAM)
The Stack Manages function calls, local variables, and return addresses. Read / Write Volatile (RAM)
The Heap Reserved for dynamic memory allocation at runtime (e.g., malloc). Read / Write Volatile (RAM)

The Stack

The Stack is a "Last-In, First-Out" (LIFO) structure. Think of it as a literal stack of plates; you can only interact with the one on top.

  • Growth Direction: On most architectures (including x86 and MSP430), the Stack grows downwards from high memory addresses to low addresses.
  • The Stack Pointer (SP): A dedicated register that always holds the address of the last item added to the stack.
  • Primary Functions:
    • Function Calls: When a subroutine is called, the CPU pushes the Return Address (the address of the instruction after the call) onto the stack.
    • Local Variables: Variables defined inside a function are temporarily stored here.
    • Register Preservation: Before a function modifies a register, it can PUSH the original value onto the stack and POP it back later to restore the state.

Stack vs. Heap: The Crucial Differences

As a researcher, you must distinguish between these two because memory corruption in the Stack usually leads to Control Flow Hijacking, while Heap corruption leads to Data Piracy or Arbitrary Execution.

Feature The Stack The Heap
Management Automatic (managed by the CPU/Compiler). Manual (managed by the programmer/OS).
Access Speed Extremely fast (uses dedicated registers). Slower (requires pointer indirection).
Structure LIFO (Orderly and predictable). Unstructured (Leads to fragmentation).
Lifespan Variables die when the function returns. Variables persist until explicitly freed.
Failure Mode Stack Overflow (too many nested calls). Memory Leak (failing to free memory).

Let's visualize that with an example:

#include <stdlib.h>

int global_var = 42;          // Data Segment

void myFunction(int param) {  // param is on the Stack
    int local_var = 10;       // Stack
}

int main() {
    int* heap_ptr = malloc(8); // heap_ptr (the variable) is on the Stack
                               // The 8 bytes allocated are on the Heap
    myFunction(5);
    return 0;
}

The stack starts at the "top" (the highest possible address the OS gives the program) and grows toward the center. The heap starts at the "bottom" (just above the code) and grows "up" toward the center.

graph TD subgraph Memory_Space ["Process Memory Map"] direction TB High["High Memory Addresses (e.g., 0xFFFF)"] subgraph Stack ["The Stack (Grows DOWN ⬇️)"] direction TB S1["Frame: main()
(contains 'heap_ptr')"] S2["Frame: myFunction()
(contains 'param' & 'local_var')"] S3["Return Address
(Points back to main)"] end Gap["'The Void' (Free Memory)"] subgraph Heap ["The Heap (Grows UP ⬆️)"] direction BT H1["malloc(8)
(8 bytes of dynamic data)"] end Data["Data Segment
(global_var = 42)"] Text["Text/Code Segment
(Machine Code of myFunction & main)"] Low["Low Memory Addresses (e.g., 0x0000)"] end %% Connections showing growth High --- Stack Stack --> Gap Heap --> Gap Gap --- Data Data --- Text Text --- Low %% Styling for clarity style Stack fill:#f96,stroke:#333,stroke-width:2px style Heap fill:#6cf,stroke:#333,stroke-width:2px style Text fill:#dfd,stroke:#333 style Data fill:#ffd,stroke:#333
When analyzing a binary, the Stack is your primary target for finding "Return-Oriented Programming" (ROP) chains. If you can overwrite a return address on the stack, you can force the program to jump to any code you choose.

Now that we know where data lives (Memory/Stack), we need to discuss the tools that actually manipulate it: The Registers.

Registers are your "workbench." Data is pulled from memory into a register, modified by the ALU, and then pushed back.

The CPU Workbench: Registers & Status Flags

To understand a disassembled program, you must track how data flows through these high-speed storage units.

1. General Purpose Registers (GPR)

In x86/x64, these are your "multi-tools." While they can hold almost anything, they have traditional roles, here are a few examples:

Register (64-bit) Register (32-bit) Name Role
RAX EAX Accumulator Stores return values from functions.
RCX ECX Counter Used as a loop counter in string or repetitive operations.
RDX EDX Data Often holds I/O pointers or overflow from math operations.
RBX EBX Base A general pointer to data in the data segment.

2. The Special Purpose Registers (The Navigators)

These define the execution state of the program. If you control these, you control the machine.

  • RIP (Instruction Pointer): Points to the next instruction to be executed. In RE, we look at this to see where the program is headed.
  • RSP (Stack Pointer): Points to the top of the stack. It changes with every PUSH or POP.
  • RBP (Base Pointer): Anchors the current "Stack Frame." It helps the CPU find local variables relative to a fixed point.

3. The Status Register (EFLAGS)

The Status Register (SR) is a collection of single-bit "flags" that describe the result of the last calculation. These are the "decision makers" for jumps (JZ, JNZ, etc.).

  • Zero Flag (ZF): Set to 1 if the result of an operation is exactly zero.
    • RE Context: When you see a CMP EAX, EBX followed by a JE (Jump if Equal), the CPU is actually checking if the ZF is set to 1.
  • Negative Flag (NF/SF): Set if the result is negative (the Most Significant Bit is 1).
  • Carry Flag (CF): Set if an unsigned operation has an overflow.
  • Overflow Flag (OF/VF): Set if a signed math operation exceeds the register's capacity.
graph LR subgraph CPU [Internal CPU Logic] direction TB REG[Registers: RAX, RBX] ALU[ALU: Arithmetic Logic Unit] SR[Status Register: ZF, CF, OF] end MEM[(Memory/Stack)] -- "1. Fetch Data" --> REG REG -- "2. Perform Calculation" --> ALU ALU -- "3. Update Flags" --> SR ALU -- "4. Store Result" --> REG REG -- "5. Push Back" --> MEM
In the MSP430 architecture I mentioned, the status bits are part of R2. In x86, they are in the EFLAGS register. Same concept, different names; this is how you spot "logic" in a sea of hex.

An Assembly Cheat Sheet: Data & Control Flow

A more complete cheatsheet can be found here for full reference, yet for now let's deal with some of the most important concepts in Assembly.

1. Data Movement: Moving the "Stuff"

These instructions move data between registers and memory.

Instruction Full Name Action RE Context
MOV dst, src Move Copies src to dst. The bread and butter of all code.
LEA dst, [src] Load Effective Address Calculates the address of src and puts it in dst. Often used for math (e.g., LEA EAX, [EBX+8]) or getting pointers to local variables.
PUSH src Push Places src onto the Stack and decrements the Stack Pointer (SP). Used to pass arguments to functions or save register states.
POP dst Pop Takes the top value off the Stack, puts it in dst, and increments SP. Used to clean up the stack or restore registers after a function call.

2. Arithmetic: The "Math"

These commands perform the work and, crucially, update the Flags (ZF, CF, SF, OF).

  • ADD / SUB: Basic addition and subtraction.
  • INC / DEC: Increments or decrements a register by 1.
  • CMP dst, src: The most important for RE. It performs a subtraction (dst - src) but does not save the result. It only updates the Flags.
    • Example: If EAX is 5 and EBX is 5, CMP EAX, EBX results in 0, so the Zero Flag (ZF) is set to 1.

3. Control Flow: The "Decision Makers"

This is where the program "thinks." Jumps rely entirely on the state of the Status Register.

Unconditional Jumps

  • JMP <addr>: Directly changes the Instruction Pointer (RIP/PC) to the target address. No conditions asked.
  • CALL <addr>: Pushes the Return Address (the address of the next instruction) onto the Stack and then jumps to the function.
  • RET: Pops the Return Address from the Stack and puts it back into the Instruction Pointer, returning control to the caller.

Conditional Jumps (The "Logic" Gates)

These only jump if the specific Flag in the Status Register is set.

Instruction Condition Logic (Flags) Use Case
JE / JZ Jump if Equal / Zero ZF = 1 Triggered after a CMP where both values were identical.
JNE / JNZ Jump if Not Equal / Not Zero ZF = 0 Triggered if the values were different.
JG / JL Jump if Greater / Less Varies Used for signed comparisons (e.g., if (x > -5)).
JA / JB Jump if Above / Below CF based Used for unsigned comparisons.

Visualizing a Logical Branch

Here is how an if-else statement looks in "Assembly Logic":

graph TD
    Start[CMP EAX, 0x10] --> Decision{Zero Flag set?}
    Decision -- "Yes (ZF=1)" --> Target[JE: Jump to Admin_Access]
    Decision -- "No (ZF=0)" --> Continue[Continue to User_Access]

    style Decision fill:#f9f,stroke:#333
When reverse engineering, look for the CMP followed by a JZ or JNZ. This is often a security check. By changing a single byte in the binary (e.g., changing a JZ to a JNZ), you can flip the logic of the program; this is known as patching.

Part 01 Conclusion: Connecting the Dots

We have covered the "why," the tools, the memory landscape, and the CPU's internal logic. To wrap up the first part of this series, let's look at a concrete Assembly Analysis. This is the "Aha!" moment where the Stack, Registers, and Instructions finally collide.

A small exercise: Assembly Breakdown

Imagine we are looking at a disassembled binary here. We see this snippet:

0x401080:  PUSH EBP            ; 1. Save the old Base Pointer on the Stack
0x401081:  MOV  EBP, ESP       ; 2. Set the new Base Pointer to the current Stack Pointer
0x401083:  SUB  ESP, 0x10      ; 3. Move Stack Pointer DOWN by 16 bytes (Space for local vars)
0x401086:  CMP  DWORD [EBP-4], 0 ; 4. Compare a local variable (at EBP-4) to Zero

The Analysis:

  • Lines 1-2 (The Prologue): This is standard housekeeping. The CPU is setting up a "Stack Frame" so the function has its own workspace.
  • Line 3 (Allocation): By subtracting from ESP, the program "claims" 16 bytes of memory on the Stack. In RE, this is a signal: "This function uses local data."
  • Line 4 (The Check): It looks at a specific spot on the stack (EBP-4) and compares it to 0. Depending on the result, the Zero Flag (ZF) in the Status Register will flip, determining which way the program jumps next.

The Journey Continues

This is just the beginning of this series. We've successfully mapped out the anatomy of a program and the "workbench" of the CPU.

In Part 02, we will stop looking at static code and start moving.. how exciting, isn't it?

Stay curious, keep poking at the bytecode, and I'll see you in the next part of the journey.
Subscribe to my monthly newsletter

No spam, no sharing to third party. Only you and me.

Member discussion