Programming structures in assembly code

While reversing, sometimes it is good to know what high-level construction this piece of code belongs to. There are several constructions which can be expressed in assembly language, if-code, switch-code, loop-code(for, while, do). These exist in all(presumably) high-level languages.

You can check examples I made here to this simple C code:

#include <stdio.h>

int main(){
    int in;
    scanf("%d\n", &in);

    if(in == 0)
        printf("Input is 0\n", in);
    else if(in == 111)
        printf("Input is 111\n", in);
    else if(in == 22)
        printf("Input is 22\n", in);
        printf("Input is unknown\n");

        case 0:
            printf("Input is 0\n", in);
        case 111:
            printf("Input is 111\n", in);
        case 22:
            printf("Input is 22\n", in);
            printf("Input is unknown\n", in);

    for(int i = 0; i < in; i++){
        in ^= i;
        in ^= i;

    int in_copy = in;
    while(in_copy != 0){
        in_copy *= 2;
        in_copy /= 2;

        in_copy += 3;
        in_copy -= 2;
    } while(in_copy != in);

When you encounter simple condition in code, program checks if condition is true or false with one of instructions changing bits in EFLAGS. One of the bits is set to 1 or 0 to indicate corresponding boolean value. Most of the times instruction is either TEST or CMP, but it can also be AND, XOR, and others. After condition is checked, conditional jump instruction is executed. Most likely it is either JZ(JE) or JNZ(JNE, these mnemonics are the same), but it depends on what condition was in source code. Majority of programs frequently check for equailty/inequality or whether a variable contains zero or not. The last thing to remember is program often inverses the meaning of conditional jump instruction. If you had condition like (var == 0), compiler would choose JNZ, not JZ, to jump to code which gets executed if condition is false. This is if-code layout:

The code layout changes a bit when else branch is present:

Now the compiler pushes in additional jump instruction. If this instruction didn’t exist in code, then program would execute else-code after doing if-code.

Things get more complex if source code contains several else-if branches:

For every if-block, there’s corresponding condition checking code, if condition is true, code continues execution and jumps from the body of condition code. Otherwise, program does conditional jump to the other checking block when checking condition. This is going on until program covers all conditional blocks altogether.

Switch construction code is similar to if-else-if code. But it can apply optimization to code execution flow.

Compiler can insert condition in case there is no matching values so that program jumps to default branch, evading other checking blocks. If-else-if code would not do this.

For-loop code layout looks this way:

The same is true for while-loop. First part is initialization code for local loop variables, like index. After that program jumps to condition code, which checks if loop should continue running. If condition is not true, loop is over.

Do-while loop is simpler because it uses only one jump:

The Visual Studio compiler uses different layout for while-loop and for-loop. I tested it with Visual Studio 2010, though this is old, I suppose modern Microsoft compilers produce the same code. 

Remember what I told you about inversing of jump instruction condition in the beginning, it seems the same rule applies to VS compiler as well.

Special purposes of some registers

This one is very small topic.

EAX - stores result of the function for virtually any calling convention ECX - used as counter by some x86 instructions(LOOP, JCXZ), stores pointer to object for some implementations of thiscall convention EDX - complement of value stored in EAX(CDQ would store EAX’s MSB in EDX, MUL would store its result in pair EDX:EAX) ESI and EDI - used as source and destination registers accordingly, by instructions SCASB, STOS, MOVS.

Application Binary Interface

ABI (Application Binary Interface) is how program communicates with other binary modules. Any decent program written in C/C++, would be linked against libc(C library). Libc has to interact with underlying operating system to carry out tasks it should perform(interacting with files, reading input and writing output, exiting program). The interaction mechanism used by libc is system calls. System call is like a function called in usermode(programs not having same priviliges as kernel), but gets redirected in kernelmode, which is kernel code. System calls can be considered as API to interact with OS kernel. System calls technique is used for convenience of other programs to implement their functionality. Also, it draws out explicit boundary between kernelmode and usermode, which is important for operating system security.

System calls are implemented with processor interrupts. An interrupt is an event processor has to take care of immediatelly after it has been emerged. For x86 compatible processors, there is Interrupt Descriptor Table(IDT). This data structure points to privileged code which will be executed for corresponding interrupt. IDT is set up by operating system when it is loading. There are three types of interrupts: hardware interrupts, exceptions, and software interrupts. Hardware interrupts are generated by computer’s hardware only(mouse, keyboard, ethernet adapter) to handle the specified event. Exceptions are made by processor to indicate special system state: divide-by-zero error, breakpoint hit, memory access violation, even processor chip overheat. Software interrupts are called via INT instruction, and used for system call mechanism. Program can also use this mechanism via SYSENTER/SYSCALL. SYSENTER is available only for 32-bit processors, programs compiled in 32-bit code normally use only INT instruction instead of this. SYSCALL is used only by 64-bit processors, this instruction replaces INT.

System call mechanism is one of the things you should know about ABI for modern operating systems. The other one is how actual functions are called within executable binary after it gets compiled and linked with existing libraries. This ABI’s part is called calling conventions. I’ll review following convention formats: cdecl, stdcall, fastcall and thiscall. This list is so short compared to how many existing calling conventions there are because there are 2 major compilers we will deal with(GCC and MSVC). They primarily use only these conventions.

When program uses a function, it calls it with CALL instruction. Before it executes this instruction, program passes arguments for function to call. Any program written for either Windows NT systems or Linux distributions would pass arguments in 2 major ways: via stack or registers. If program uses stack to store arguments, all arguments are pushed in right-to-left order for any calling convention, i.e. the first argument to be pushed is the last argument of the function.

Calling conventions are distributed between 2 different groups depending on what function cleans up the stack. Cdecl is caller-cleanup convention, which means it is caller who is responsible for freeing stack space occupied by arguments. You can see it by instruction ADD ESP, n right after CALL instruction. Stdcall is opposite, this format requires callee function to clean the stack arguments. If you see RET instruction with immediate value argument, or ADD ESP, n right before RET in callee, probably this is stdcall function.

Fastcall convention is spin-off of stdcall, it passes 2 first arguments in ECX and EDX. Thiscall is something you see only in programs compiled from OOP languages, like C++. Thiscall functions are(as expected) non-static class member functions, which pass “this” object via either register or stack, depending on architecture and compiler.

Things are different if program is compiled for x64 OS. The first several arguments are passed in registers, the others are located in stack. For Windows, the first four arguments are placed in RCX, RDX, R8, R9. For Linux, the first six arguments are RDI, RSI, RDX, RCX, R8, R9.