Updates on HEIR, the Homomorphic Encryption Compiler Project

On 2026-08-14 I published an article connected the Google Security blog with an update connected HEIR, our homomorphic encryption (HE) compiler. This is simply a companion article, in which I person nary limits connected connection count aliases jargon, and I tin consciousness free to beryllium honest. So strap in.

Assuming you won’t publication the linked firm blog post, HEIR is simply a compiler that converts an input programme to a programme that operates straight connected encrypted data. The guarantee of homomorphic encryption is that, assuming you haven’t cracked the cryptography, at nary constituent does the machine moving the program get moreover a azygous spot of accusation astir the cleartext data used to make the encrypted inputs. No accusation astir the inputs, outputs, or immoderate intermediate values.1

The blog station focuses connected HEIR’s ability to compile pre-trained ML models, and gives 4 examples of small, but nontrivial models that it tin compile. Hence, homomorphic encryption tin alteration services to supply perfectly backstage inference. I’ll effort to opportunity much astir erstwhile and wherever this is useful later successful this article. First I wanted to springiness a much actual sense for really HEIR useful successful the discourse of these examples, and outline (my position on) the project’s roadmap for the future. I won’t do a heavy dive connected HEIR’s internals by immoderate means, since that would make the article excessively long. Give maine a outcry if you want that, but location are plentifulness of docs to publication done astatine heir.dev and you tin spot a caller (fast-paced) talk I gave astatine ASPLOS this year.

Table of Contents:

The repo down the blog post

The blog station ends pinch a database of examples compiled pinch HEIR. Those examples constituent to a GitHub repository2 that you tin clone and tally yourself. The biggest hurdle is installing bazel, and past bazel hermetically manages everything else.3

Some elemental runtime comparisons

The simplest and fastest illustration to effort is the credit paper fraud detector. This is simply a elemental three-layer feed-forward web pinch sigmoid activations, trained connected a Kaggle dataset. The linear layers person dimensions 128, 64, and 2 (the past being the logits for the 2 classes, fraud and not-fraud).

You tin tally the basal illustration successful 1 line:

This bid will compile the (pre-trained, checked-in) cc_fraud model to the Lattigo backend, and then run it connected a sample input. The bid supra outputs:

The cardinal constituent present is that the information of the exemplary connected encrypted inputs took astir 2 seconds connected a single-threaded CPU.

Compare this to the aforesaid execution connected cleartext inputs, noting that this is the latency of a azygous inference, truthful it doesn’t use from amortization. (This requires fetching and encoding the original dataset, which is explained successful the README; I’ll skip that portion here).

Anyone who has heard of HE whitethorn person heard that it is slow, but I want to pause here to comparison this (single-threaded CPU, non-amortized!) runtime: 2 seconds for HE conclusion vs 0.5 sclerosis for cleartext. This is simply a 4,000x slowdown, and the computation involves 2 matrix-vector products (where the matrix is not private), pinch 2 evaluations of a sigmoid function.

There are galore caveats to this demo worthy concisely noting:

• This exemplary is mini capable that the backstage accusation fits successful a single (CKKS) ciphertext. Inputs tensors pinch much than, say, 32k elements will require aggregate ciphertexts and people much overhead.
• This exemplary is mini capable that bootstrapping (the slowest portion of HE) is not required.
• I did not see a simulated web overhead, nor did I see the per-user one-time setup required to make and upload the applicable key material that makes HE work.
• The server is doing a decent magnitude of (one-time, model-specific) pre-computation successful the “Running preprocessing” step.

The different examples successful the repo are much complex, and hence person longer latencies and worse overhead vs cleartext (and a representation request of 60-90 GiB). In particular:

• network_anomaly, an ensemble of auto-encoders: 30 seconds for inference.
• criteo, a recommender exemplary adapted specifically for HE: 5 minutes for inference.
• hotword, a 10-layer convolutional network: 20 minutes for inference.4

This sounds bad, but retrieve it’s single-threaded CPU execution. Our colleagues moving to merge HEIR pinch GPUs person reported that the criteo workload runs successful ~500ms connected a azygous GPU (similar to an H100). Compare that to the criteo/cleartext:evaluate_cleartext demo which runs successful 10ms, and you’re down to a 50x slowdown (again, the baseline is non-amortized CPU execution). That activity wasn’t capable to make it into the Google blog post, but they are compiled by HEIR pinch immoderate pending upstream PRs. My constituent is that the execution times are continuing to improve, and for immoderate mini problems they could beryllium called reasonable if you squint. And this does not moreover breach the taxable of HE accelerated by FPGAs and ASICs, which are moreover much promising performance-wise.

So alternatively of the Google firm blog station showcasing earthy performance, it was meant to showcase the expressiveness of HEIR: it tin compile a batch of models, and the capacity connected immoderate of them is decent.

As acold arsenic showcasing features, the repo besides shows really 1 mightiness usage HEIR to:

• Target and comparison aggregate HE backends (the repo has Lattigo and OpenFHE as examples, effort bazel tally -c opt //demos/cc_fraud/openfhe:evaluate_fhe.
• Explore per-layer timing: HEIR has the expertise to insert debugging callbacks into the compiled program, effort bazel tally -c opt //demos/cc_fraud/lattigo:evaluate_fhe_timing. In that bid you’ll spot the first linear furniture takes the mostly of the runtime, ~1.2 seconds retired of 2 seconds total.
• Explore an incorrect inference: Similar to timing, the debug callback can decrypt ciphertexts and inspect them for correctness aliases precision loss. Try bazel tally -c opt //demos/cc_fraud/lattigo:evaluate_fhe_debug and you’ll see output that shows that by the extremity of the inference, the usage of HE has caused about 2 bits of precision nonaccomplishment compared to the cleartext model. This precision nonaccomplishment tin beryllium tuned pinch compiler flags and trades against performance. Unfortunately, navigating that trade-off requires HE expertise.

Getting a exemplary into HEIR

The process of getting a pre-compiled exemplary to beryllium thing that HEIR tin process is not yet automated. The main 2 constraints are:

1. You request to beryllium capable to person the pre-compiled exemplary to MLIR, which is the intermediate practice that HEIR uses to correspond programs. Many ML frameworks such arsenic PyTorch and JAX person devices to person to MLIR.
2. You request to manually annotate your programme pinch HEIR-specific annotations that opportunity (a) what inputs to the conclusion are concealed and (b) what are bounds on the input ranges to each activation function.

The demos successful the repository show really to do this for PyTorch, and I’m moving pinch the maintainers of torch-mlir to add a feature that will alteration maine to automate this (given a validation group to usage to estimate ranges).

That said, overmuch of the early stages of the compiler pipeline (recognizing activations, fusing linear layers, etc.) is based connected really torch-mlir exports models to MLIR, truthful if we want to adhd ONNX aliases JAX support (both person awesome MLIR exporters), it will apt not activity retired of the container conscionable yet. Moreover, we don’t moreover person complete sum of torch operators yet. A batch of the specifications of really to support linear algebraic operators successful HE are both tricky and progressive investigation topics (for MLIR enthusiasts, we don’t support linalg.generic successful afloat generality).

Invoking the compiler

This is simply a spot of a tangent, but if you look astatine the build rules for the examples successful the repo, you’ll spot immoderate somewhat messy calls to a macro that invokes the compiler.

What’s going connected present is that HEIR’s interface is simply a batch much like LLVM than clang. There are 2 binaries, heir-opt and heir-translate, which grip moving compiler passes and codegen, respectively, matching LLVM’s opt and translate. Those 2 binaries are wrapped into bazel rules, which I published as rules_heir, and past further wrapped the rules successful macros that correctly stitch together the optimizer and codegen binaries, and wrap the results into a cc_library, go_library, or rust_library, arsenic due to the chosen backend.

One bully facet of this is that rules_heir uses pinned binaries, and tin beryllium pointed to a nightly aliases civilization HEIR release. Still, this is little of a clear personification communicative than thing for illustration clang, which has a accordant interface and constricted vulnerability of the underlying LLVM room sink.

Our extremity is to yet make 1 (or multiple) clang-style products that person a narrower scope (e.g., only supporting 1 frontend) and conceptually simpler flags (like -O2). For frontends for illustration JAX/PyTorch, this will astir apt be wrapped successful a Python library.

Speaking of Python libraries! We do person a heir_py frontend library. It’s wholly unrelated to ML astatine this point. It really compiles a (very limited) subset of Python bytecode (using numba’s frontend) to homomorphic encryption, and past runs HEIR, generates OpenFHE C++ code, compiles and links that pinch clang (the slowest part!), and past loads the instrumentality codification backmost into a Python module which the personification tin telephone for illustration a function.

To beryllium clear, we haven’t utilized the Python frontend for thing peculiarly useful. It still needs a batch of features to commencement being useful, such arsenic preserving numpy ops arsenic MLIR linalg ops that HEIR understands.

Who wants HE?

Now I’d for illustration to return a measurement backmost and pontificate a spot astir who wants HE. There has been a batch of chat successful the HE organization astir this, and successful peculiar the mobility has been raised: what is HE’s “killer app”?

Most academics will show you the reply is “private LLMs.” I will reserve my sentiment connected that taxable for meal parties, but suffice it to opportunity that 2 things are presently true: First, astir if not each applied HE activity successful academia is presently focused on making encrypted LLM conclusion faster. Second, the SOTA is still very acold distant from being practical. In particular, latencies are measured successful the units of seconds per token, and each solutions I’m alert of require a round-trip to the client to decrypt the output from 1 conclusion and hole it for the adjacent conclusion step. And this is including hardware acceleration (e.g., 8 GPUs). I’m besides not an master connected LLM architectures, truthful I don’t person a strong sense for the existent barriers and really apt they are to succumb to further investigation efforts.

My attraction astatine Google has alternatively been connected a lower-hanging fruit: what applications tin return advantage of the HE capabilities we person today? In that discussion, I person landed connected the pursuing soft heuristics, which shed immoderate ray connected the remaining challenges to make HE genuinely practical. I will again restrict myself to backstage conclusion here.

Models from 2020

While the latency of HE has improved significantly, models pinch billions of parameters are still retired of reach for applicable applications. And successful particular, this rules out any transformers that are large capable to beryllium useful.

That said, I person spoken pinch engineers and scientists astatine Google who person informed maine that, contempt their efforts, their domain has NOT yet seen immoderate use from applying transformers. As such, the halfway complexity of their exemplary architectures has not changed importantly since astir 2020.

Convolutions are still king successful galore domains. Or successful different words, location are still useful tasks that don’t need immense models. And truthful moreover if a ample exemplary could beryllium used, if privateness is important enough, a smaller 1 mightiness beryllium acceptable to get accelerated HE.

Critical privateness for some parties

While I aspire to a world successful which HE tin beryllium applied without a captious request for privacy, we’re not location yet. So for HE to beryllium useful, the privateness of some parties must beryllium critical.

I opportunity “both parties” for the pursuing reason. The modular privateness protection 1 expects erstwhile discussing backstage inference is delicate personification data. If my email work can’t spot the matter of my emails, I americium spared from the consequence of information leaks, insider threats, exposure to warranted searches, etc. The modular measurement to protect personification privateness successful these situations is to do each the activity locally connected the user’s device. In precocious 2024, Google Maps changed its location history feature to beryllium afloat on-device, at slightest partially successful consequence to US constabulary geofence warrants.

But for backstage conclusion successful particular, the trained exemplary is besides delicate information of the work provider. Shipping a exemplary to a device, moreover 1 pinch a unafraid enclave, risks vulnerability of the exemplary weights to competitors. For usage cases for illustration biometric authentication, leaking the exemplary besides adds a information risk; attackers tin technologist attacks against the accumulation model in an unrestricted environment. I’m not an master connected hardware-based security, but immoderate of my colleagues make a surviving breaking them, and the measurement they talk astir the topic doesn’t animate assurance successful unafraid enclaves.

I person besides been led to judge that location are usage cases where keeping a proprietary exemplary concealed exceeds the importance of personification privacy, because the personification information successful those situations is not considered sensitive. In these situations, “user privacy” is simply a bonus on apical of protecting IP. That’s not arsenic romanticist arsenic personification privacy, but hey, if it spurs further finance successful HE, I’ll put up pinch it to get person to the ideal.

Not excessively galore simultaneous users

As latency becomes little of a bottleneck for HE, other bottlenecks people emerge. The biggest 1 I spot is cardinal management.

The mathematics that makes HE work requires typical “evaluation keys.” These keys are, rather literally, encryptions of the user’s secret cardinal (or thing derived from the user’s concealed key). Ignoring for a infinitesimal the added presumption of circular security, what matters for applicable HE systems is:

1. Evaluation keys are unsocial to each personification of the service.
2. They standard pinch the programme size successful immoderate consciousness (e.g., doing larger matmuls successful HE requires much information keys than a smaller matmuls).
3. The biggest examples from the HEIR demo repository request ~40 GiB of evaluation keys. SOTA LLMs successful HE request hundreds of GiBs of cardinal material.

So if you had a million-user work pinch the 40-GiB-per-user usage case, you’d request 512x20T difficult drives conscionable to shop the cardinal material. At ~500 USD per thrust (just checked Amazon, truthful that’s a user price), you’re asking for 250k USD other retention costs. I’m judge the value of RAM needed to service a decent QPS is going to beryllium worse.

And more, if you are utilizing GPUs to accelerate the HE, you person the added problem of getting this 40 GiB of cardinal material from disk to the GPU erstwhile a personification makes a request. My napkin mathematics says this should return roughly hundreds of milliseconds–the aforesaid bid of magnitude arsenic the entire HE computation. And during the clip that you’re filling up the GPU’s memory pinch keys for 1 user, you can’t service different personification requests.

To beryllium fair, a decent chunk of the RAM request consists of preprocessed model weights, which are model-dependent and not user-specific. And moreover, astir of the cardinal worldly required is to support bootstrapping, so location whitethorn beryllium bully HE applications (like Apple’s) that tin usage tricks to debar needing to bootstrap. Maybe successful the champion lawsuit we’d only request 1 GiB of cardinal material. So possibly you could fresh capable keys for 50 users connected a beefy datacenter GPU. Still, you tin spot the systems bottlenecks here.

And I don’t deliberation this is getting overmuch better. Newer HE schemes that committedness amended latency thin to require larger key material, not smaller. As such, until we (the HE community) finds a measurement to trim the memory requirements of HE by an bid of magnitude, I americium only going to beryllium looking for applications wherever location aren’t too galore simultaneous users.

The server shouldn’t request to spot the output

A batch of group I’ve talked to who want to usage homomorphic encryption eventually recognize that, actually, the server wants to study 1 spot of accusation astir the user’s underlying cleartexts. For example, opportunity we want to authenticate a fingerprint without having to shop the user’s biometrics. HE would forestall the server from seeing the authentication result, and truthful it can’t enactment connected that.

One basal thought for dealing pinch this is to nonstop the encrypted authentication result back to the user, person the personification decrypt it, and nonstop the consequence backmost to the server. The evident downside to this is that you can’t spot the client to honestly study the authentication result!

To make this work, you would request the client to make a zero-knowledge impervious that the value they decrypted was really the consequence of moving the decryption routine with the provided ciphertext and the user’s concealed key. It has to beryllium zero-knowledge because the server cannot study thing astir the user’s concealed key. While I haven’t done this myself, I person heard it is some imaginable and location are still immoderate challenges in scaling it to HE schemes for illustration CKKS. And of course, it further eats distant astatine a latency budget.

As such, near-term applications of HE are astir apt going to have an easier clip making it to accumulation if the application does not require the server to spot the computation result.

So what applications meet these constraints?

Some ideas that look to clasp water:

Remote diagnostics. Say you tally a mill and you person immoderate expensive machine you bought from a manufacturer. You want to fto the manufacturer diagnose the instrumentality to urge attraction and place problems, but the specifications of what you’re utilizing the instrumentality to make (say, manufacturing volume) aliases the settings of the instrumentality itself would beryllium somewhat sensitive financial accusation if it were leaked to a competitor. However, diagnostic reports are batch jobs that the server request not spot the output of, there aren’t millions of factories, diagnostics are a premium work the shaper would want to complaint for, and they are astir apt not that analyzable of models.

I person heard rumors that personification is really utilizing HE for this, but I person no publicly verifiable proof. Our web intrusion discovery demo is similar in nature, wherever the contents of the packets are the delicate data.5

B2B analysis: Similar to existing applications of backstage group intersection at Google, business-to-business applications cheque a batch of boxes. Both businesses are privacy needs, but location are only 2 users, truthful cardinal worldly is not an obstacle. But there is capable spot for the 2 parties to stock the output, provided the computation is producing immoderate group of business metrics applicable to some parties. And business analytics and metrics reporting are usually rather latency tolerant. I callback talking to 1 HE practitioner who said, “if the customer only needs a study generated erstwhile a month, past who cares if the HE computation takes a week to run?”

Specialized biometrics situations: I deliberation biometric applications for illustration face and fingerprint nickname are wrong the capabilities of existent HE techniques, pinch the objection of the cardinal worldly problem. But location are specialized situations successful which other latency should beryllium tolerated, specified as in-person relationship creation aliases relationship betterment procedures (think of a caller worker on-boarding day, aliases losing your information badge). These events are infrequent capable that the strategy wouldn’t beryllium encumbered by high QPS needs. And I deliberation it is mostly a bully thought that employers do not have databases of worker biometrics.

Some notes connected the HEIR roadmap

I’ll adjacent pinch a fewer notes connected wherever HEIR is going next.

Crypto successful the compiler

You whitethorn person noticed each the HEIR backends are presently outer room APIs in a precocious level language. This is chiefly because, erstwhile we started HEIR, hardware acceleration activity was still successful its infancy. Moreover, group building accelerators (and GPU libraries) needed an easy way to constitute larger applications that usage their libraries for benchmarking purposes. These often mirrored immoderate existing room API (like OpenFHE), and past injected their ain codification astatine immoderate level, specified arsenic scheduling a key switching op connected a GPU.

So we decided HEIR should meet them successful the middle. Start by targeting these celebrated room APIs, and get the frontend of the compiler pipeline moving well. At the aforesaid time, we person been slow building retired a existent “backend” of the compiler, which involves implementing the HE cryptography directly as compiler passes. With that successful place, and pinch our multi-layer design enabled by MLIR, we tin compile a programme down to a room API (high-level HE scheme operations), aliases to a lower-level modular polynomial arithmetic which immoderate accelerators usage arsenic their starting point, or to an moreover lower-level vectorized modular arithmetic level (think of a GPU aliases TPU for 64-bit integer mathematics modulo civilization primes, alternatively of bfloat16) which immoderate accelerators usage arsenic their starting point, or moreover further down to LLVM, and past to x86 aliases wasm aliases whatever.

This will beryllium captious for aggregate of our hardware partners, including Niobium, Cornami, and Optalysys, all of whose hardware we are readying to target pinch HEIR. This way is making dependable progress, but I expect it to prime up given that we person some a batch much GPU activity maturing, and that I personally should have clip successful the adjacent 6 months to really excavation into that work. That will successful move unfastened a batch of imaginable for optimizations that impact scheduling, kernel fusion, etc.

GPU and TPU support

We person a bunch of parallel efforts astatine integrating TPU and GPU backends into HEIR (still arsenic room APIs for now). This includes the CHEDDAR library, Belfort’s GPU room (I’m not judge if their task codename is public), and CROSS/jaxite. We besides person heard liking from a variety of other groups moving connected GPU acceleration.

In my view, GPU/TPU will beryllium important to getting HE successful production before civilization ASICs tin beryllium scaled up successful datacenters.

New schemes

Researchers are coming retired pinch caller schemes each fewer years, and HEIR is meant to support each of them. Some that are connected our radar include

• Gentry-Lee which is optimized for matrix multiplication and has a batch of imaginable for GPU/TPU synergies.
• Poulpy which streamlines and simplifies a batch of the underlying polynomial mathematics of CKKS, and also makes switching betwixt schemes overmuch simpler. This has already had some progress.
• Gao-Zheng which uses a fancy encoding to alteration some arithmetic and logical style operations successful 1 scheme.

Benchmarking

One of the large trading points of HEIR is benchmarking. In particular, I want to beryllium capable to benchmark each the different hardware solutions, HE schemes, and optimization techniques so that I tin beryllium informed capable to determine what typical hardware Google should merge into its datacenters (when the clip comes).

As portion of that my colleague Shruthi Gorantala has been difficult astatine work aligning the manufacture astir a benchmarking solution successful the shape of fhe-benchmarking.org.

We want to adhd features to HEIR truthful that it tin make codification successful the right structure to beryllium submitted to the benchmarking repository. This would make it very easy for group moving connected HEIR (particularly researchers) to trial and showcase the benefits of their optimizations.

Research

A immense portion of the tie of HEIR is that researchers tin instrumentality their research ideas straight successful HEIR, aliases astatine slightest usage HEIR arsenic a frontend. So a large portion of my occupation is to thief train postgraduate students connected moving pinch HEIR, reviewing their code, etc.

Though HEIR still doesn’t person a due publication yet, it has been racking up citations, and, to my count, 4 published papers were based connected investigation done straight successful HEIR.

Join us!

If this interests you, aliases if you want to get immoderate acquisition in compilers/MLIR aliases cryptography, HEIR is afloat unfastened source, and we person some a play agency hours and a (recorded) monthly meeting where we reappraisal activity that happened successful the past period and talk upcoming design work. Everyone is welcome, but we conscionable inquire that you show up to 1 of these in-person meetings (or schedule a backstage gathering pinch a maintainer) before submitting a PR, which is portion of our policy to trim AI spam.