Solo – a .so loader for static Linux binaries

Ship 1 musl-linked executable. At runtime, load the user's existing glibc-linked GPU driver. No container, nary AppImage, and nary 2nd libc successful the process.

Static binaries are a wonderfully boring measurement to deploy package connected Linux: one file, nary dependencies, thing to break. We build ours with IX, a source-first build strategy for producing fully fixed Linux binaries. The boredom ends the infinitesimal the exertion needs the GPU: Vulkan and OpenGL drivers are supplied by the big arsenic shared objects, usually built against glibc, and a afloat fixed musl binary cannot normally dlopen() them.

SoLo crosses that boundary. It provides a dlfcn-style root API backed by its ain ELF loader (x86-64 and aarch64) and a glibc ABI span implemented on top of musl. The consequence is still 1 mean fixed executable, but it tin usage the graphics driver already installed connected the machine.

The repository includes an end-to-end Vulkan proof: a afloat fixed executable loads the host's unmodified Vulkan driver, runs a compute shader, and writes the consequence to a PNG. Tested connected AMD radv, radeonsi, Intel, and NVIDIA GPUs under Linux, and connected Apple M1 nether Asahi Linux.

The big keeps the hardware-specific code. You vessel everything else.

And not connected a demo's connection alone: connected each commit, CI loads the shared libraries of the 1,000 most-installed Debian packages — complete 2,100 host objects — done SoLo, connected some x86-64 and aarch64.

Grab the prebuilt binary — nary clone, nary toolchain, immoderate Linux pinch a Vulkan driver installed (mesa-vulkan-drivers is enough):

vulkan-aarch64 is the aforesaid demo for arm64 machines. The bid discovers the distro-installed Vulkan ICD successful the accustomed measurement and produces a 512×512 RGBA image. This is really we build the Shitty merchandise binaries—a blazingly fast terminal emulator, BTW! To unit a peculiar driver:

ICD manifest names alteration somewhat betwixt distributions. Passing nary --driver lets the embedded Khronos loader execute its normal discovery.

You tin verify that the executable itself is not dynamically linked:

Or build the aforesaid demo from source, pinch Python 3 and a C/C++ compiler in PATH:

This is not a artifact telephone to vkCreateInstance. The demo:

1. enters the statically linked Khronos Vulkan loader;
2. loads the host's Vulkan ICD and its non-glibc limitations done SoLo;
3. creates a device, retention buffer, descriptor set, and compute pipeline;
4. dispatches a checked-in SPIR-V shader;
5. maps the consequence and writes it done statically linked libpng.

The complete illustration is successful bin/vulkan, and the Vulkan program itself is successful main.cpp.

elf_loader.cpp maps ELF segments, walks DT_NEEDED, resolves versioned symbols, applies x86-64 relocations, supports ELF TLS and TLSDESC, materializes IFUNCs, applies RELRO, and runs initializers. Dependencies that are themselves ELF DSOs are loaded recursively.

glibc is deliberately not loaded. Imports specified arsenic malloc@GLIBC_2.2.5 are resolved by glibc_shim.cpp to ABI-correct adapters over the process's existing musl runtime. Unsupported glibc functions person unique generated stubs that neglect loudly pinch the nonstop awesome and type if they are ever called, alternatively of silently corrupting the process.

Because musl sizes its synchronization objects to the glibc ABI of each architecture, the span does not protector them: a pthread_mutex_t a driver creates is utilized successful place. A fastener is truthful 1 fastener for some the loaded DSO and the fixed executable that whitethorn stock it, and glibc's fixed recursive and error-check initializers are adopted connected first use.

Before loading a DSO from disk, SoLo checks its fixed supplier registry. This lets an exertion fulfill a dependency—Wayland, for example—with functions already linked into the executable. LD_LIBRARY_PATH and DL_ELF_LIBRARY_PATH are honored for libraries extracurricular the modular system directories.

The absorbing pieces are mini capable to read:

• lib/dlfcn.cpp — dlopen, dlsym, errors, and fixed providers
• lib/elf_loader.cpp — ELF mapping, symbols, relocations, and TLS
• lib/glibc_shim.cpp — implemented glibc ABI adapters
• lib/glibc_stubs.cpp — definitive fallbacks for the remainder of the ABI

The default target builds the standalone archive:

The published ./dlfcn symlink points to the resulting libdlfcn.a. Include lib/dlfcn.h, nexus the archive into a musl-static application, and mean dlopen()/dlsym() calls are redirected to SoLo. The root tree is intentionally self-contained and suitable for copying into different static build graph.

CI performs the autochthonal build and trial connected Alpine/musl pinch GCC, Fedora pinch GCC, and Ubuntu pinch Clang. The Vulkan trial installs each distribution's own Lavapipe package; it does not tally the driver from an Arch sysroot.

Every build input for the standalone Vulkan executable is vendored under bin/vulkan. build.py compiles those sources directly: upstream CMake, Meson, configure, and Make build systems are not invoked.

• musl 1.2.5 (0784374d561435f7c787a555aeab8ede699ed298)
• LLVM runtimes 15.0.7: libc++, libc++abi, libunwind, and compiler-rt builtins (8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
• Vulkan Headers 1.4.357 (e3b1eec08173d6b825cd3ac88c885a63b621504a)
• Vulkan Loader 1.4.357 (5f157b62e333c63260d05d81bf66faa216ab0fb8)
• zlib 1.3.2 (da607da739fa6047df13e66a2af6b8bec7c2a498)
• libpng 1.6.50 (2b978915d82377df13fcbb1fb56660195ded868a)

License files are retained beside the corresponding sources. shader.inc is the checked-in SPIR-V shape of shader.comp, truthful nary shader compiler is required.

In the wide case, only SoLo lets a fixed exertion show the dynamic loader: "for this strategy DSO's libwayland dependency, usage the symbols already linked into my executable." This lets the exertion embed the newest libwayland alternatively of targeting the oldest type disposable connected every supported system.

And the bound betwixt the 2 worlds is not a bladed dlsym shim — it carries the parts that make overseas codification really behave:

• C++ exceptions transverse it successful some directions. A propulsion successful the fixed world unwinds done glibc-compiled frames into a glibc catch, and the other way around, destructors moving connected some sides: the guests' _Unwind_* imports are bound to the 1 unwinder successful the executable, truthful location is a single objection machinery successful the process alternatively of 2 fighting ones.
• All 4 TLS models, without wrappers aliases codification patching. General- and local-dynamic done __tls_get_addr, TLSDESC done its custom-ABI resolver, and initial-exec — whose GOT slots are plain thread-pointer-relative offsets nary loader tin intercept — served from a surplus arena that rides successful the executable's ain fixed TLS, truthful one process-wide offset is valid successful each thread and unmodified musl does the per-thread layout.
• ld.so's binding semantics, not an approximation. Global-scope interposition, RTLD_DEEPBIND, DT_SYMBOLIC, awesome versioning pinch the unversioned-provider compatibility rule, lazy PLT binding pinch the argument registers preserved done the resolver, GNU and SysV hash lookups, ifunc resolvers handed their hwcaps, /etc/ld.so.cache.
• Cross-world introspection. backtrace() walks fixed and glibc frames alike and names some done 1 dladdr; dl_iterate_phdr, dladdr1, and the link_map destruction fto unwinders and profilers spot each image; the file-backed mappings support existent paths successful /proc/self/maps for debuggers.
• The stateful corners of glibc, for real. getcontext / makecontext / swapcontext successful assembly against glibc's mcontext layouts connected some architectures, the pre-2.34 pthread ABIs, GNU obstacks, the fortified _chk family, and the inline-stdio ABI — musl's FILE is deliberately laid retired truthful glibc's inlined putc_unlocked compiles against it — down to _IO_2_1_stdout_ resolving to musl's ain stream.

Every 1 of these is exercised by a conformance artillery compiled against real glibc headers astatine -O2, and by loading each shared entity of the thousand most-installed Debian room packages successful CI, connected x86-64 and aarch64.

• gcompat is simply a distribution-level glibc API shim for moving prebuilt glibc binaries connected musl. Its loader stub re-executes the programme done musl's move linker pinch libgcompat.so preloaded; utilizing it from a musl programme requires linking that shared library or adding it to the loaded DSO's DT_NEEDED. It does not springiness a afloat static musl process a move loader. SoLo's self-contained exemplary is stronger: the executable embeds some the ELF loader and ABI bridge, loads unchanged host DSOs without a strategy compatibility package, preserves the versions of their glibc imports, and lets unused unsupported functions stay behind symbol-specific, fail-loud stubs alternatively of blocking the full DSO.
• Detour bootstraps the system's ld-linux and allows aggregate C runtimes to coexist. SoLo takes the opposite route: it maps the required DSOs itself and translates their glibc imports onto musl, truthful a 2nd libc and its TLS authorities ne'er participate the process.
• Cosmopolitan Libc's cosmo_dlopen() follows the aforesaid split-runtime strategy arsenic Detour, pinch each of its advantages and drawbacks: it bootstraps the host's ELF expert and libc, then delegates loading the target DSO to the host's dlopen().
• ClickHouse's experimental userspace move loader currently maps ELF objects itself, but stops short of loading glibc. Its proposed way to real-world strategy libraries specified arsenic CUDA is Detour-like: bootstrap the system's ld.so, support a 2nd libc runtime, and switch the musl/glibc thread pointer astatine each boundary. SoLo alternatively implements the glibc ABI complete the host's musl runtime and tin fulfill DSO limitations from providers already linked into the fixed executable.
• graphics.gd's musl + dlopen experiment follows the aforesaid split-runtime exemplary arsenic Detour: an embedded helper brings in the host's glibc loader, and assembly trampolines move betwixt musl and glibc TLS astir overseas calls. This leaves 2 independent TLS worlds: every boundary crossing needs a trampoline, and a callback implemented successful musl cannot beryllium passed safely to glibc codification because glibc invokes it while its own TLS is active. SoLo keeps a azygous musl TLS world instead.
• Flatpak, AppImage, and containers lick the problem by hiding a mini Linux distribution wrong aliases astir your program. This useful successful astir the same way that moving location solves a missing powerfulness adapter. The consequence is simply a huge blob afloat of duplicated libraries, mounts, namespaces, extraction tricks, and runtime indirection—all of which make profiling, debugging, and basic introspection worse. Shipping a distro because you request 1 strategy .so is not portability. SoLo ships 1 normal, inspectable executable and borrows the only constituent that genuinely belongs to the host: its hardware driver.

• Linux only, connected x86-64 and aarch64. The loader, the TLSDESC and lazy-PLT resolvers, and the initial-exec arena screen both; the glibc symbol inventories are generated per architecture, truthful printf@GLIBC_2.2.5 connected one is printf@GLIBC_2.17 connected the different without a azygous translator norm in the code;
• focused connected existent Mesa/Vulkan ICD dependency closures, and driven by the top 1000 Debian room packages by popcon votes: the 885 of them that ship glibc-linked shared objects — astir 2100 objects — each load done SoLo in CI connected some architectures. Loading is the floor, not the claim: calls into the symbols the span still stubs abort loudly, and dev/abi-demand.txt is the remaining work, classed by how galore installations request each symbol;
• a load-once runtime (dlclose succeeds but does not unload an image);
• supporting each 4 TLS models. Initial-exec variables are placed successful a 16 KiB surplus arena that rides successful the executable's ain fixed TLS, truthful one process-wide offset is valid successful each thread without patching musl. The one restriction: threads created before a dlopen spot zero-initialized TLS for the modules it loaded, truthful load initial-exec libraries earlier spawning the threads that usage them. An initial-exec module that does not fresh the arena fails to load pinch an correction naming the image and the byte counts;
• explicit astir missing ABI coverage: an unimplemented glibc telephone aborts and names itself.

The extremity is to move the difficult wall betwixt “fully static” and “uses the system GPU” into a finite, testable compatibility layer. The Vulkan PNG is the first proof that the wall has a door.