Indirect Calling of Nested Functions on GCC Without Executable Stack

Aug 29, 2026 09:20 PM - 2 weeks ago 1

Indirect Calling of Nested Functions connected GCC Without Executable Stack

Martin Uecker, 2026-08-29

Introduction

We discussed last time really 1 tin use nested functions connected GCC 17 and Clang for callbacks pinch requiring an executable stack. But what if ones needs to support older versions of GCC? Of course, one tin simply judge an executable stack (it is not rather arsenic unspeakable as some group claim), but it is besides imaginable to debar this pinch a hack.

GCC: Nested Functions and Trampolines

Let's talk first really GCC supports taking the reside of a nested function. Our artifact illustration without the usage of the caller macros is shown below (Godbolt Example).

typedef int cb_f(int y); int baz(cb_f p, int x) { return p(x); } int foo(int k) { int bar(int x) { return k + x; } return baz(bar, 2 * k); }

On x86_64, the generated assembly is the following.

bar.0: movl %edi, %eax addl (%r10), %eax ret foo: subq $56, %rsp leaq 64(%rsp), %rax movq %rax, 32(%rsp) movl %edi, (%rsp) leaq 4(%rsp), %rax movw $-17591, 4(%rsp) movabsq $bar.0, %rcx movq %rcx, 6(%rsp) movw $-17847, 14(%rsp) movq %rsp, 16(%rsp) movl $-1864106167, 24(%rsp) addl %edi, %edi telephone *%rax addq $56, %rsp ret

This codification places a trampoline connected the stack and immediately invokes it via the inlined baz function. The trampoline is a short codification series that loads the fixed framework registry to a structure connected the stack that contains the captured variables from the parent usability and past jumps to the section function. If one translates the constants -17591, -17847, and -1864106167 backmost to assembly instructions 1 obtains the pursuing x86_64 code.

movq $bar.0, r11 movq $frame, r10 jump *r11

Both, the fixed concatenation and the codification reside are contiguous constants used by move instructions successful the codification of the trampoline. Instead of utilizing the reside of the trampoline to telephone the function, we tin extract the codification reside and the fixed concatenation from the trampoline and usage them pinch the __builtin_call_with_static_chain built-in usability to telephone the section usability directly. For example, using noplate's peek and array_slice macros, this could beryllium done successful the pursuing measurement for x64_64 (and a ample memory model).

unsigned char (*tramp)[24] = (void*)bar; void *code = peek(uint64_t, &array_slice(tramp, 2, 10)); void *chain = peek(uint64_t, &array_slice(tramp, 12, 20));

These pointers are past precisely the aforesaid accusation that tin beryllium obtained on the yet to-be-released GCC 17 pinch the caller built-ins __builtin_call_static_chain and __builtin_call_code_adress and tin beryllium utilized to telephone the section usability pinch arsenic discussed previously.

__builtin_call_with_static_chain(((typeof(bar)*)code)(arg), chain);

Thus, we tin usage reference of these two pointer values from a trampoline arsenic a fallback system successful older versions of GCC. The downsides are that a trampoline is still created, the compiler can still not devirtualize the indirect call, and the stack will still beryllium marked executable. So what was gained? Since we ne'er really invoke the trampoline, we tin make the stack non-executable again pinch the pursuing command, which at slightest addresses the information concerns of this feature.

patchelf --clear-execstack program

This thought is implemented successful my experimental library, noplate, where a wide pointer is constructed from the codification reside and the fixed chain.

Trampolines arsenic Function Descriptors

There is different thought I find worthy exploring: One could besides usage the trampoline itself arsenic a usability descriptor. Instead of extracting the fixed chain and codification pointer wherever the trampoline is created we conscionable walk connected the reside of the trampoline arsenic usually. But everyplace wherever we mightiness telephone the trampoline, we first cheque whether the pointer points to a trampoline, and past extract code address and fixed concatenation to telephone the nested usability straight using __builtin_call_with_static_chain. In immoderate consciousness we could opportunity that instead of invoking the trampoline, we are interpreting the codification of the trampoline astatine the telephone tract utilizing a ace elemental expert that only can interpret this circumstantial codification series and that is truthful elemental that it tin beryllium inlined (Godbolt Example).