Intercepting DLL libraries calls. API hooking in practice

API hooking methods for programmers by using DLL libraries forwarding mechanism (DLL proxy).

Sometimes we need to intercept certain DLL library calls, we might discovered an application bug or we want to add an extra feature to the application or to log the invoked functions and its parameters. In normal conditions we have access to the source codes and function modification is just a matter of source code editing, but sometimes we just don't have access to the source code of the library or the software, like in many cases isn't distributed with the source code. What to do in this case? In this article you can read about popular API hooking solutions, and there will be presented slightly different approach to this topic.

API Hooking

The most common solution that probably most of you know is called API Hooking, a technique which consists in the fact that the libraries function calls redirect to your code. Most popular API hooking libraries are Microsoft Detours (usef frequently for game hacks), but the price tag on this commercial library is set to 9,999.95 USD (around 31737 PLN!), for Delphi we can find madCodeHook library, its price is € 349 for commercial usage. Beside mentioned libraries, there are many other and free libraries.

API hooking for loaded DLL libraries and their functions works by patching (overwriting) first bytes of the function prologue code we want to hook with a JMP NEAR instruction to our code, encoded in hex as E9 xx xx xx xx). It looks like this:

After the control is passed back to our function, usually we can run our own code, run the original function and return back to the code that invoked the original function from the DLL library.

API Hooking can cause several problems, it's all related to the structure of the compiled applications and the structure of its code, problem occurs when we would like to invoke original function from the hook itself (usually this would end as an infinite loop), in those cases it's necessary to create a special code chunk aka trampoline that allows to invoke original function code, despite the hook itself in the function body.

API Hooking technique is practically impossible to use in case of protected DLL libraries, when every change to the library code on the disk or in the memory is not possible when for example CRC checks are present etc.

Classic API Hooking is also not suitable for intercepting pseudofunctions exported by the DLL libraries, I'm talking about exported variables, class pointers etc., because in this type of exports there's no way to create a classic code hook between the original function and our intercepting code (there's no function to hook at all). This kind of problems can be solved with PE (Portable Executable) export table modification, but its less popular solution and very few hooking libraries even supports it.

DLL Forwarding

One of the creative and more troublesome ways of API hooking in DLL libraries is using internal Windows mechanism called DLL Forwarding, basically it means forwarding DLL calls to another module.

This technique is based on using replacement library, so called proxy DLL, it exports all of the original library functions and passes all of the calls to the original library except for the functions we want to hook. Function calls are passed to the original library by using barely known Windows mechanism, that lets to use other library functions like they were stored in the hooking library, but in fact their code is located in other library, that's why the name DLL forwarding - from forwarding, redirection.

Function calling convention

Function calling convention is a low level way of passing parameters to the functions and stack handling mechanism before the function return. Mostly it depends on the compiler settings and in most of the high level programming languages it's possible to change the calling convention to whatever you want, either by changing the compiler settings or by using special programming language constructs (pragmas etc.). In order for our hooking library to work correctly, its hooking functions has to use the same calling convention as the hooked functions, they just have to be binary compatible in other case it might end with an exception caused by stack damage etc.

Table 1.Function calling conventions.

Calling conventions highly depends on the compiler default settings, and for example Delphi uses register calling convention by default, for the C programming language cdecl is the default calling convention.

WinApi functions (Windows system functions) uses stdcall calling convention, so before the call, the parameters are stored on the stack using push instructions, then the call instruction is executed, after the call there's no need to correct the stack pointer ESP, because in stdcall convention the stack is automatically corrected just before the function returns. It's interesting that some of the WinApi functions aren't using stdcall calling convention but the cdecl, where the parameters are stored on the stack, but the stack correction has to be done after the call by the compiler, based on the number of the parameters passed to the function. One function that uses this convention is wsprintfA() from the Windows system library USER32.dll (its counterpart in C libraries is sprintf()), this way of calling functions was probably introduced, because it's impossible to tell how many parameters were actually used by the function itself (only compiler knows this).

API hooking example

As an example we are going to use our test library BlackBox.dll, it exports only two, sample functions Sum() and Divide(), as you can guess the first function adds two numbers and the second function divides two numbers. Lets assume we have a full documentation of this library and we know what the calling convention is used for those two functions (assume we have header files for this library) and we know what parameters are used. In other cases we would have to use reverse engineering to obtain this kind of low level information.

Listing 1.Functions description from BlackBox.dll library

// this function adds two numbers and stores the result
// in „Result” variable and returns TRUE on success
// FALSE is returned on error
BOOL __stdcall Sum(int Number1, int Number2, int * Result);

// this function divides two integer numbers and stores the result
// in „Result” variable and returns TRUE on success
// FALSE is returned on error
BOOL __stdcall Divide(int Number1, int Number2, int * Result);

In our example library, the Divide() function is buggy, and dividing by zero causes an exception and our application crashes (lets assume our application doesn't handle exception handling). Our goal is to fix this problem.

Proxy DLL

To fix the buggy function in BlackBox.dll library, we're going to create intermediary library, with valid Divide() function implemented, that can handle division by zero without causing an exception. Implementation will be coded using 32 bit assembler, using FASM compiler (polish assembler compiler, created by mr Tomasza Grysztar). Below you will find sample library template with precise code structures comments.

Listing 2.Beginning of our library.

;-------------------------------------------------
; DLL output file format
;-------------------------------------------------
format PE GUI 4.0 DLL

; DLL entry point function name in our library
entry DllEntryPoint

; include file with Windows functions and constants
include '%fasm%\include\win32a.inc'

Here you can find output file declaration type, at the beginning of the source code also header files can be placed as well as the name of the entry point function for the DLL library.

Listing 3.Uninitialized data section

;-------------------------------------------------
; uninitialized data section
;-------------------------------------------------
section '.bss' readable writeable

; uchwyt HMODULE oryginalnej biblioteki
	hLibOrg		dd ?

Executable files as well as DLL libraries are divided in sections, one of them is a section with uninitialized data, it doesn't take any space on the disk, but only holds the information about the total size of the uninitialized variables application is using. Section names in executable files doesn't matter (it's only limited to 8 chars), usually contractual names are used and in the section declaration access rights has to be defined (read, write, executable), but in FASM compiler case .bss section declaration, creates an uninitialized section for the variables.

Listing 4.Data section

;-------------------------------------------------
; initialized data section
;-------------------------------------------------
section '.data' data readable writeable

; name of the original library
	szDllOrg	db 'BlackBox_org.dll',0

Here we have a name of the original library, that was renamed to BlackBox_org.dll (it's stored in the source code in ASCIIz format, null terminated), it's going to be used in the further code so it can be loaded.

Listing 5.Code section with DLL entry point.

;-------------------------------------------------
; library code section
;-------------------------------------------------
section '.text' code readable executable

;-------------------------------------------------
; DLL library entry point (DllMain)
;-------------------------------------------------
proc DllEntryPoint hinstDLL, fdwReason, lpvReserved

	mov	eax,[fdwReason]

; event send right after the DLL library is loaded
	cmp	eax,DLL_PROCESS_ATTACH
	je	_dll_attach

	jmp	_dll_exit

; library was loaded
_dll_attach:

; get the handle to the original DLL library, it can
; be used if we wish to call the original functions
	push	szDllOrg
	call	[GetModuleHandleA]
	mov	[hLibOrg],eax

; return 1, that means our library initialization
; was successful
	mov	eax,1

_dll_exit:

	ret

Code section contains all of the library functions and the DLL entrypoint function, this is a special function that is called by the Windows operating system, after the library is loaded. Code section has to be marked with executable flags, this is an information to the operating system, that this memory area contains the executable code, if it wasn't marked as executable any code execution attempt from this memory area would end up as an exception on the CPU processors with DEP (Data Execution Prevention) memory protection mechanism. Inside the initialization function (DllMain), after receiving the DLL_PROCESS_ATTACH event we are using original DLL library name to get its handle, so called HMODULE (so it can be used later on to call the original functions etc.).

Listing 6.Over-optimalization protection

; call any original library
; BlackBox_org.dll function, without it, FASM compiler
; removes the reference to the library and it won't be
; automatically loaded
	call	dummy

Our library uses the original library, but if we don't put any reference to it in the source code, FASM compiler will remove any reference to it (optimization) and it won't be automatically loaded, that's why here we put a fake call to any of its function, directly after the ret instruction (so it won't be executed at any point).

Listing 7.Valid Divide() function code

;-------------------------------------------------
; our implementation of Divide() function with
; fixed division by zero handling
;-------------------------------------------------
proc Divide Number1, Number2, Result

; check the divisor for 0 value, if so
; return with the error code
	mov	ecx,[Number2]
	test	ecx,ecx
	je	DivisionError

; load first number into EAX register
	mov	eax,[Number1]

;extend EDX register by the number's sign +/-)
	cdq

; now EDX:EAX pair holds the 64 bit number

; perform division of EDX:EAX / ECX, division is
; executed on the pair of registers EDX:EAX, which
; are treated like a 64 bit number, the result
; of division is stored in EAX registed, the remainder
; is saved in EDX register
	idiv	ecx

; check for the valid pointer to the result value
; if it's not provided, return with an error code
	mov	edx,[Result]
	test	edx,edx
	je	DivisionError

; store result of division under the provided value address
	mov	[edx],eax

; return with exit code TRUE (1)
	mov	eax,1

	jmp	DivisionExit

; division error, return FALSE (0)
DivisionError:

	sub	eax,eax

DivisionExit:

; return from the division function
; exit code of BOOL type is set in the EAX register
	ret
endp

Our Divide() function implementation checks for the division by zero condition and if the divisor is set to zero, function returns the error code FALSE, additionally the pointer to the result variable is extra checked for NULL values and if its an empty pointer, error code is returned as well. Also please notice the calling convention of the function is exactly the same as the calling convention used by the original function, and in our case stdcall convention is used, so the parameters are passed on the stack, function return value is stored in the EAX register

Listing 8.Import table of our library.

;-------------------------------------------------
; section with functions used by our library
;-------------------------------------------------
section '.idata' import data readable writeable

; a list of libraries we use in our code
library kernel,'KERNEL32.DLL',\
	blackbox, 'BlackBox_org.dll'

; a list of functions from the KERNEL32.dll library
import	kernel,\
	GetModuleHandleA, 'GetModuleHandleA'

; here we declare the usage of the original library
; DLL library will be automatically loaded
import	blackbox,\
	dummy, 'Divide'

FASM compiler lets us manually define libraries and functions used by our library, beside standard system libraries, we need to add here a reference to the original BlackBox_org.dll library. Thanks to this, Windows while loading our hooking library will also load the original library in our address space and we won't have to manually load the original library with the LoadLibraryA() function call. In some cases it's even mandatory to load library using import table entry, it's required for dynamic link libraries that uses TLS (Thread Local Storage) mechanism used by multithreaded applications.

Listing 9.Exported functions table

;-------------------------------------------------
; export table section with functions exported
; by our library, here we must also declare all of
; the functions declared in the original library
;-------------------------------------------------
section '.edata' export data readable

; a list of exported functions and its pointers
export	'BlackBox.dll',\
	Sum, 'Sum',\
	Divide, 'Divide'

; name of the forwarded function, first the name
; of the destination library is stored (without the .DLL extension)
; after the period the name of the final function is saved
	Sum db 'BlackBox_org.Sum',0

In this section we must declare all of the functions from the original library, functions that we want to hook must be implemented in the code, functions that we want to pass to the original library are stored in a special text format:

DestinationDllLibrary.FunctionName

or

DestinationDllLibrary.#1

for the ordinal exported function, not by the name. All of the inner working of this mechanism is handled by the Windows itself, thus DLL Forwarding.

Listing 10.Relocation section

;-------------------------------------------------
; relocation section
;-------------------------------------------------
section '.reloc' fixups data discardable

Last section in our library is the relocation section, it's required for proper working of our library. It's because dynamic DLL libraries can be loaded in various base address spaces by the Windows and absolute addresses used for pointers and assembler instruction using absolute addresses has to be updated accordingly to the current base address in the memory and this information is generated in the relocation section by the compiler.

Summary

This API hooking method can be used with success to modify any application that uses dynamic DLL libraries for their working. It has its pros and cons (in relate to the classic API Hookingu methods), but in my opinion it opens much wider space for experiments and offers much easier way of changing complete functionality of the application. Implementation of this method can also be done in high level programming languages with proper usage of the export functions definition files (DEF).

References