We eliminated 1,400 CVEs in NanoClaw's container images

Last week we announced Echo's business pinch NanoClaw, designed to widen the imagination and information of the unfastened root project. In this post, we want to propulsion backmost the curtain and show you precisely really Echo's agentic hardening process works.

How do we observe CVEs?

Before we tin hole anything, we request a complete, trustworthy image of what's really successful the image. We scan and analyse the upstream NanoClaw instrumentality utilizing respective independent vulnerability scanners, including Trivy, Grype, and Wiz. Here's the earthy consequence of scanning the unfastened root NanoClaw image pinch Grype, sorted by severity:

And here's really that stacks up against comparable supplier runtimes (Hermes and OpenClaw) crossed some Grype and Trivy (we besides added the NanoClaw Echo image to this comparison - which we will dive into soon):

Now, let’s move to the fixing and CVE reduction.

Step 1: Start pinch what we tin safely bump

Every room successful the image is its ain problem to solve. So the first point we do is abstracted the findings into "safe to bump" and "needs existent work." The easy wins are the libraries we cognize we tin upgrade without breaking NanoClaw. Chromium is simply a awesome example. It's good known for backward compatibility, truthful we tin spot their updates and bump pinch confidence. What we are near pinch are the CVEs that can’t beryllium fixed, and the ones pinch awesome jumps. Once we portion retired the Chromium-related CVEs, we're still near pinch astir 600 vulnerabilities that request fixing. So what happens next?

Step 2: The bumps that require existent research

Some upgrades require a awesome type jump, which will apt not activity retired of the box. In these cases, we person to spot it ourselves and verify the spot really holds without breaking the app. See below, a actual example: Hono's node-server

On paper, this looked for illustration a awesome jump. But erstwhile we dug into the root code, we recovered the hole was disposable wrong a overmuch person type to what was installed, type 1.19.14 (even if the scanners’ vulnerability databases weren't alert of yet). As portion of our publication to the unfastened root community, we added it to their advisory (it’s a process we do connected a regular ground successful Echo). We move connected to the adjacent measurement - the “won’t fix” ones. 

Step 3: Patching and Backporting

Then you deed the wall: the remainder of the findings that are marked arsenic won't hole aliases that the distro maintainers only hole successful caller majors, which arsenic mentioned supra are apt to break your app. For these, the hole strategy of prime is to backport, which intends taking a spot from a newer type of the package and applying it to the older type that the app requires. In our case, that intends we find the hole successful the latest upstream version, and commencement moving connected NanoClaw’s root codification directly. There are 3 main challenges we request to equilibrium between:

• Finding the correct fix - knowing wherever the bug is, tracing the fixing commit, and confirming the hole is genuinely safe and complete - not each hole sources are safe to use, truthful further investigation is needed.
• Applying it without breaking anything - the spot has to beryllium compatible pinch the existing app cleanly.
• Validating - compatibility, functionality and that the CVE was genuinely resolved.

On apical of the exertion dependencies, there's the operating strategy underneath everything. NanoClaw's Dockerfile builds connected Debian 12

Debian 12 guidelines images bring a agelong tail of OS-level vulnerabilities pinch them. This is wherever Echo OS comes into play. It's the Linux distro that Echo maintains, and it's compatible pinch the communal upstream distros - Ubuntu, Debian, RHEL, Amazon Linux, and more. Every portion of it is built from root truthful it tin beryllium continuously patched by our AI patching agents. It offers thousands of patched OS packages and has eliminated much than 1.1 cardinal CVEs crossed them.

How Echo conducts backporting

Let’s drawback 1 of the latest backports we did arsenic portion of the NanoClaw project. We’ll attraction connected CVE-2025-59375 successful expat. This process is conducted by Echo’s proprietary backporter agent.

Why did we prime CVE-2025-59375?

It's the hardest benignant of backport location is - a information hole that isn't a bounds cheque but a caller subsystem threaded done the mediate of the library, carried from a newer upstream merchandise down into the older type we ship, and 1 wherever the upstream maintainer explicitly warned distributors not to effort a partial cherry-pick.

How tin CVE-2025-59375 beryllium exploited?

An attacker sends a small, wholly well-formed XML archive and the parser allocates a wildly disproportionate magnitude of heap. Upstream's ain figure: a ~250 KiB archive caused astir 800 MiB of allocation - an amplification facet of astir 3,300. The consequence is representation exhaustion and process death, aliases an OOM termination that takes neighbours pinch it.

Why is it difficult to do a backport fix?

expat already had protection against amplification attacks - the billion-laughs defence, which enforces a 100x limit erstwhile output passes 8 MiB. But that accounting measures parsed output bytes, nonstop positive entity description . It ne'er looked astatine the existent heap. So you tin enactment comfortably wrong the entity fund while the parser's soul structures - hash tables, drawstring pools, DTD scaffolding - balloon. he hole had to present existent allocation accounting, and that's invasive:

•  Every soul allocation now goes done `expat_malloc` / `expat_free` / `expat_realloc`, which prepend a `size_t` header to each artifact truthful that free and realloc cognize really overmuch to subtract. That changes the pointer handed backmost to callers. Mix a earthy allocation pinch a tracked free anyplace and you corrupt the heap.

•  Structures that utilized to transportation the memory-function array now transportation the parser grip instead, because the antagonistic lives connected the guidelines parser. That cascades into signature changes crossed `dtdCreate`, `dtdReset`, `dtdDestroy`, `dtdCopy`, `hashTableInit`, `poolInit`, `copyString`.

•  `parserCreate` gains a parent-parser statement truthful external-entity kid parsers complaint the guidelines parser's budget, pinch strict initialisation ordering - immoderate fields must beryllium group earlier immoderate tracked allocation tin happen.

•  Three categories of allocation must deliberately bypass tracking: the app-facing `XML_MemMalloc`/`XML_MemRealloc`, the main input buffer from `XML_GetBuffer`, and the contented exemplary handed to the element-declaration handler (applications free that 1 pinch plain `free()`). Get immoderate of them incorrect and you either corrupt representation aliases commencement rejecting morganatic documents.

What did we do and what yet worked?

The upstream hole was written against a newer expat than the 1 we ship, truthful this is simply a backport from a higher type down to a little one, and it is ample successful each direction. Upstream's alteration spans 17 files; ours ends up rubbing 9, crossed the parser core, the nationalist headers, the CLI tool, the archiving and the trial suite. The reference spot does not use to our tree, and landing only the parts that hap to fresh produces precisely the authorities upstream warns astir - immoderate allocations tracked and immoderate not, which is worse than not patching astatine all. What worked was treating it arsenic a rebase alternatively than an apply. Our supplier runs the full point extremity to end: apply, build, tally the afloat upstream trial suite, diagnose each nonaccomplishment into a circumstantial class, manus it to the fixer built for that class, and repetition until the build and the full trial suite pass. The classes are things for illustration a hunk whose statement numbers nary longer match, surrounding discourse that has changed, a record that has moved aliases been split, codification that nary longer compiles against this version, and tests that neglect aft a cleanable compile - each pinch its ain dedicated fixer.

Three things had to beryllium solved for this spot specifically:

1. A renamed build guard. The hole wraps the caller locator successful `#ifdef XML_DTD`; successful our type that macro was renamed to `XML_GE` backmost successful 2.6.0. Keep the aged sanction and the preprocessor silently drops the full hole - greenish build, passing tests, scanner sees a patched version, vulnerability still present. This is the astir vulnerable nonaccomplishment mode successful the full job, because thing tells you.

2.  A re-organized trial suite. Upstream had divided its monolithic `runtests.c` into per-area files, truthful the patch's trial changes had to beryllium redistributed into `alloc_tests.c` and `nsalloc_tests.c`. That's why our spot touches 9 files wherever the reference touches 8.

3.  Tests that break because the hole is correct. This is what drove america to adhd a dedicated test-fixing capability. Two existent examples: an existing allocation trial asserted that parsing survives a definite number of failing reallocations - nary longer existent erstwhile realloc routes done the tracked allocator, truthful the assertion needed adapting alternatively than deleting. And 1 of the caller tests raises the amplification limit to beryllium the limit is enforced; the changeless that useful upstream doesn't activity successful our tree, because our parser has allocated a different magnitude by that constituent successful the test.

That past class is wherever it's tempting to cheat, truthful the test-fixer operates nether difficult rules: it whitethorn only modify the CVE spot itself, it whitethorn ne'er weaken, skip aliases disable a test, and immoderate adapted trial must still validate the information fix. Fixing the build by silencing the trial suite is precisely the result we're trying to make impossible.

‍The last spot is 9 files, 64 hunks, +786/-112, pinch the complete upstream trial suite passing.

The extremity product

On the near broadside you tin spot the reference fix, and connected the right, Echo’s backporting supplier suggested a fix. This CVE had 3 steps, the 1 beneath is the renaming of the XML_DTD characteristic defender to XML_GE successful 2.6.0.

Most of the activity is done by our triaging agent, pinch our engineers near to do the last reappraisal and approval.

Step 4: Mirroring and syncing

Once our mill verifies the results and confirms that each disposable upstream fixes person been applied, the image is built and pushed into the Echo store, and past synced backmost to NanoClaw. That’s an important note: Echo only implements charismatic fixes. We don’t create civilization patches to guarantee exertion compatibility and safety, minimal drift from upstream, and a trustworthy process for devs utilizing Echo OS. 

The mirroring

The Echo shop tin beryllium mirrored to different registries, and successful our lawsuit to a dedicated registry for NanoClaw. When a caller image is published, it's automatically synced to the NanoClaw ECR wrong minutes. From that constituent on, each image is continuously monitored for recently fixed CVEs and re-pulled consecutive from the Echo shop to the NanoClaw registry. 

The extremity result

So really cleanable does it really get? Here's NanoClaw connected Echo versus the alternatives:

We eliminated astir 99% of the CVEs. The fistful that stay are going to support being monitored by our mill and resolved arsenic soon arsenic its possible.

That's the full loop: observe pinch aggregate scanners, bump what's safe, investigation the tricky ones, backport the "won't fix" findings ourselves, and support it cleanable done automated mirroring.

FAQ

What is CVE backporting, and why does Echo usage it?

Backporting intends taking a information spot from a newer type of a package and applying it to an older version. Echo uses it erstwhile a nonstop upgrade would require a awesome type jump that risks breaking the app. This lets Echo hole vulnerabilities marked "won't fix" while staying compatible pinch NanoClaw's existing root codification and dependencies.

Which vulnerability scanners does Echo usage to observe CVEs?

Echo scans the upstream NanoClaw instrumentality pinch respective independent scanners, including Trivy, Grype, and Wiz. Using aggregate scanners produces a much complete and trustworthy image of what's really successful the image. Echo besides benchmarks results against comparable supplier runtimes for illustration Hermes and OpenClaw to show really the hardened image compares crossed different scanning tools.

How galore CVEs does Echo really eliminate?

Echo eliminates astir 99% of CVEs versus the hundreds aliases thousands recovered successful comparable runtimes. The fistful that stay are vulnerabilities pinch nary disposable upstream fix, patch, aliases bump. Echo stays afloat upstream-compatible by creation and waits for the maintainer organization to merchandise fixes.

How does the mirroring and syncing process work?

Once Echo's mill verifies each upstream fixes, the image is packaged and pushed to the Echo store, past mirrored to a dedicated NanoClaw registry. From there, each image is continuously monitored for recently fixed CVEs and re-published consecutive from the Echo shop to support it cleanable complete time.