WebLLM: high-performance in-browser LLM inference engine

Sep 02, 2026 09:02 PM - 1 day ago 7

NPM Package "WebLLM Chat Deployed" Join Discord MLC LLM

High-Performance In-Browser LLM Inference Engine.

Documentation | Blogpost | Paper | Examples

WebLLM is simply a high-performance in-browser LLM conclusion motor that brings connection exemplary conclusion straight onto web browsers pinch hardware acceleration. Everything runs wrong the browser pinch nary server support and is accelerated pinch WebGPU.

WebLLM is fully compatible pinch OpenAI API. That is, you tin usage the aforesaid OpenAI API connected any unfastened root models locally, pinch functionalities including streaming, JSON-mode, function-calling (WIP), etc.

We tin bring a batch of nosy opportunities to build AI assistants for everyone and alteration privateness while enjoying GPU acceleration.

You tin usage WebLLM arsenic a guidelines npm package and build your ain web exertion connected apical of it by pursuing the examples below. This task is simply a companion task of MLC LLM, which enables cosmopolitan deployment of LLM crossed hardware environments.

In-Browser Inference: WebLLM is simply a high-performance, in-browser connection exemplary conclusion motor that leverages WebGPU for hardware acceleration, enabling powerful LLM operations straight wrong web browsers without server-side processing.
Full OpenAI API Compatibility: Seamlessly merge your app pinch WebLLM utilizing OpenAI API pinch functionalities specified arsenic streaming, JSON-mode, logit-level control, seeding, and more.
Structured JSON Generation: WebLLM supports state-of-the-art JSON mode system generation, implemented successful the WebAssembly information of the exemplary room for optimal performance. Check WebLLM JSON Playground connected HuggingFace to effort generating JSON output pinch civilization JSON schema.
Extensive Model Support: WebLLM natively supports a scope of models including Llama 3, Phi 3, Gemma, Mistral, Qwen(通义千问), and galore others, making it versatile for various AI tasks. For the complete supported exemplary list, cheque MLC Models.
Custom Model Integration: Easily merge and deploy civilization models successful MLC format, allowing you to accommodate WebLLM to circumstantial needs and scenarios, enhancing elasticity successful exemplary deployment.
Plug-and-Play Integration: Easily merge WebLLM into your projects utilizing package managers for illustration NPM and Yarn, aliases straight via CDN, complete pinch broad examples and a modular creation for connecting pinch UI components.
Streaming & Real-Time Interactions: Supports streaming chat completions, allowing real-time output procreation which enhances interactive applications for illustration chatbots and virtual assistants.
Web Worker & Service Worker Support: Optimize UI capacity and negociate the lifecycle of models efficiently by offloading computations to abstracted worker threads aliases work workers.
Chrome Extension Support: Extend the functionality of web browsers done civilization Chrome extensions utilizing WebLLM, pinch examples disposable for building some basal and precocious extensions.

Check the complete database of disposable models connected MLC Models. WebLLM supports a subset of these disposable models and the database tin beryllium accessed astatine prebuiltAppConfig.model_list.

Here are the superior families of models presently supported:

Llama: Llama 3, Llama 2, Hermes-2-Pro-Llama-3
Phi: Phi 3, Phi 2, Phi 1.5
Gemma: Gemma-2B
Mistral: Mistral-7B-v0.3, Hermes-2-Pro-Mistral-7B, NeuralHermes-2.5-Mistral-7B, OpenHermes-2.5-Mistral-7B
Qwen (通义千问): Qwen2 0.5B, 1.5B, 7B

If you request much models, request a caller exemplary via opening an issue aliases cheque Custom Models for really to compile and usage your ain models pinch WebLLM.

Learn really to usage WebLLM to merge ample connection models into your exertion and make chat completions done this elemental Chatbot example:

Example Chatbot connected JSFiddle Example Chatbot connected Codepen

For an precocious illustration of a larger, much analyzable project, cheque WebLLM Chat.

More examples for different usage cases are disposable successful the examples folder.

WebLLM offers a minimalist and modular interface to entree the chatbot successful the browser. The package is designed successful a modular measurement to hook to immoderate of the UI components.

# npm npm instal @mlc-ai/web-llm # yarn yarn adhd @mlc-ai/web-llm # aliases pnpm pnpm instal @mlc-ai/web-llm

Then import the module successful your code.

// Import everything import * as webllm from "@mlc-ai/web-llm"; // Or only import what you need import { CreateMLCEngine } from "@mlc-ai/web-llm";

Thanks to jsdelivr.com, WebLLM tin beryllium imported straight done URL and activity out-of-the-box connected unreality improvement platforms for illustration jsfiddle.net, Codepen.io, and Scribbler:

import * as webllm from "https://esm.run/@mlc-ai/web-llm";

It tin besides beryllium dynamically imported as:

const webllm = await import("https://esm.run/@mlc-ai/web-llm");

Most operations successful WebLLM are invoked done the MLCEngine interface. You tin create an MLCEngine lawsuit and loading the exemplary by calling the CreateMLCEngine() mill function.

(Note that loading models requires downloading and it tin return a important magnitude of clip for the very first tally without caching previously. You should decently grip this asynchronous call.)

import { CreateMLCEngine } from "@mlc-ai/web-llm"; // Callback usability to update exemplary loading progress const initProgressCallback = (initProgress) => { console.log(initProgress); }; const selectedModel = "Llama-3.1-8B-Instruct-q4f32_1-MLC"; const engine = await CreateMLCEngine( selectedModel, { initProgressCallback: initProgressCallback }, // engineConfig );

Under the hood, this mill usability does the pursuing steps for first creating an motor lawsuit (synchronous) and past loading the exemplary (asynchronous). You tin besides do them separately successful your application.

import { MLCEngine } from "@mlc-ai/web-llm"; // This is simply a synchronous telephone that returns immediately const engine = new MLCEngine({ initProgressCallback: initProgressCallback, }); // This is an asynchronous telephone and tin return a agelong clip to finish await engine.reload(selectedModel);

WebLLM supports 4 cache backends done AppConfig.cacheBackend:

"cache": browser Cache API (default).
"indexeddb": browser IndexedDB.
"opfs": browser Origin Private File System (OPFS).
"cross-origin": experimental Chrome Cross-Origin Storage API hold backend. Install the Cross-Origin Storage extension to usage it. (If the hold isn't installed, WebLLM falls backmost to the default cache automatically.)

Example:

import { CreateMLCEngine, prebuiltAppConfig } from "@mlc-ai/web-llm"; const appConfig = { ...prebuiltAppConfig, cacheBackend: "cross-origin" }; const engine = await CreateMLCEngine("Llama-3.1-8B-Instruct-q4f32_1-MLC", { appConfig, });

Notes:

If "opfs" is selected successful an situation without OPFS support, cache operations neglect pinch an OPFS readiness error.
When utilizing "opfs", appConfig.opfsAccessMode tin beryllium group to "auto" to usage OPFS sync entree handles wherever supported, aliases "sync" to require sync entree handles. The default is "async".
The "cross-origin" backend requires installing and enabling a compatible browser extension.
Cross-origin backend presently does not support programmatic tensor-cache deletion; clearing is extension-managed.

After successfully initializing the engine, you tin now invoke chat completions utilizing OpenAI style chat APIs done the engine.chat.completions interface. For the afloat database of parameters and their descriptions, cheque section below and OpenAI API reference.

(Note: The exemplary parameter is not supported and will beryllium ignored here. Instead, telephone CreateMLCEngine(model) aliases engine.reload(model) alternatively arsenic shown successful the Create MLCEngine above.)

WebLLM besides supports streaming chat completion generating. To usage it, simply walk stream: existent to the engine.chat.completions.create call.

You tin put the dense computation successful a worker book to optimize your exertion performance. To do so, you request to:

Create a handler successful the worker thread that communicates pinch the frontend while handling the requests.
Create a Worker Engine successful your main application, which nether the hood sends messages to the handler successful the worker thread.

For elaborate implementations of different kinds of Workers, cheque the pursuing sections.

WebLLM comes pinch API support for WebWorker truthful you tin hook the procreation process into a abstracted worker thread truthful that the computing successful the worker thread won't disrupt the UI.

We create a handler successful the worker thread that communicates pinch the frontend while handling the requests.

// worker.ts import { WebWorkerMLCEngineHandler } from "@mlc-ai/web-llm"; // A handler that resides successful the worker thread const handler = new WebWorkerMLCEngineHandler(); self.onmessage = (msg: MessageEvent) => { handler.onmessage(msg); };

In the main logic, we create a WebWorkerMLCEngine that implements the aforesaid MLCEngineInterface. The remainder of the logic remains the same.

// main.ts import { CreateWebWorkerMLCEngine } from "@mlc-ai/web-llm"; async function main() { // Use a WebWorkerMLCEngine alternatively of MLCEngine here const engine = await CreateWebWorkerMLCEngine( new Worker(new URL("./worker.ts", import.meta.url), { type: "module", }), selectedModel, { initProgressCallback }, // engineConfig ); // everything other remains the same }

WebLLM comes pinch API support for ServiceWorker truthful you tin hook the procreation process into a work worker to debar reloading the exemplary successful each page sojourn and optimize your application's offline experience.

(Note, Service Worker's life rhythm is managed by the browser and tin beryllium killed immoderate clip without notifying the webapp. ServiceWorkerMLCEngine will effort to support the work worker thread live by periodically sending heartbeat events, but your exertion should besides see due correction handling. Check keepAliveMs and missedHeatbeat successful ServiceWorkerMLCEngine for much details.)

We create a handler successful the worker thread that communicates pinch the frontend while handling the requests. Instantiate the handler astatine the apical level of the worker book truthful its connection listener is registered during first book evaluation. Do not instantiate it from an activate aliases connection listener: the browser tin restart an already-active worker without dispatching different activate event.

// sw.ts import { ServiceWorkerMLCEngineHandler } from "@mlc-ai/web-llm"; new ServiceWorkerMLCEngineHandler(); console.log("Service Worker is ready");

Then successful the main logic, we registry the work worker and create the motor using CreateServiceWorkerMLCEngine function. The remainder of the logic remains the same.

// main.ts import { MLCEngineInterface, CreateServiceWorkerMLCEngine, } from "@mlc-ai/web-llm"; if ("serviceWorker" in navigator) { navigator.serviceWorker.register( new URL("sw.ts", import.meta.url), // worker script { type: "module" }, ); } const engine: MLCEngineInterface = await CreateServiceWorkerMLCEngine( selectedModel, { initProgressCallback }, // engineConfig );

You tin find a complete illustration connected really to tally WebLLM successful work worker successful examples/service-worker.

You tin besides find examples of building Chrome hold pinch WebLLM successful examples/chrome-extension and examples/chrome-extension-webgpu-service-worker. The second 1 leverages work worker, truthful the hold is persistent successful the background. Additionally, you tin research different afloat task of a Chrome extension, WebLLM Assistant, which leverages WebLLM here.

Full OpenAI Compatibility

WebLLM is designed to beryllium afloat compatible pinch OpenAI API. Thus, too building a elemental chatbot, you tin besides person the pursuing functionalities pinch WebLLM:

streaming: return output arsenic chunks successful real-time successful the shape of an AsyncGenerator
json-mode: efficiently guarantee output is successful JSON format, spot OpenAI Reference for more.
seed-to-reproduce: usage seeding to guarantee a reproducible output pinch fields seed.
function-calling (WIP): usability calling pinch fields devices and tool_choice (with preliminary support); aliases manual usability calling without devices aliases tool_choice (keeps the astir flexibility).

WebLLM supports optional integrity verification for exemplary artifacts using SRI (Subresource Integrity) hashes. When the integrity section is group connected a ModelRecord, WebLLM will verify the downloaded config, WASM, and tokenizer files against the provided hashes earlier loading.

import { CreateMLCEngine } from "@mlc-ai/web-llm"; const appConfig = { model_list: [ { model: "https://huggingface.co/mlc-ai/Llama-3.2-1B-Instruct-q4f16_1-MLC", model_id: "Llama-3.2-1B-Instruct-q4f16_1-MLC", model_lib: "https://raw.githubusercontent.com/user/model-libs/main/model.wasm", integrity: { config: "sha256-<base64-hash-of-mlc-chat-config.json>", model_lib: "sha256-<base64-hash-of-wasm-file>", tokenizer: { "tokenizer.json": "sha256-<base64-hash-of-tokenizer.json>", }, onFailure: "error", // "error" (default) throws IntegrityError, "warn" logs and continues }, }, ], }; const engine = await CreateMLCEngine("Llama-3.2-1B-Instruct-q4f16_1-MLC", { appConfig, });

You tin make SRI hashes for exemplary files with:

The openssl commands require a Unix-like ammunition (macOS/Linux). On Windows, tally openssl via Git Bash aliases WSL.

If a hash does not match, an IntegrityError is thrown (or a informing is logged erstwhile onFailure: "warn"). All fields successful integrity are optional — only specified artifacts will beryllium verified. When the integrity section is omitted entirely, WebLLM behaves precisely arsenic earlier (no verification).

See the integrity-verification example for a complete moving demo.

WebLLM useful arsenic a companion task of MLC LLM and it supports civilization models successful MLC format. It reuses the exemplary artifact and builds the travel of MLC LLM. To compile and usage your ain models pinch WebLLM, please cheque out MLC LLM document on really to compile and deploy caller exemplary weights and libraries to WebLLM.

Here, we spell complete the high-level idea. There are 2 elements of the WebLLM package that alteration caller models and weight variants.

model: Contains a URL to exemplary artifacts, specified arsenic weights and meta-data.
model_lib: A URL to the web assembly room (i.e. wasm file) that contains the executables to accelerate the exemplary computations.

Both are customizable successful the WebLLM.

import { CreateMLCEngine } from "@mlc-ai/web-llm"; async main() { const appConfig = { "model_list": [ { "model": "/url/to/my/llama", "model_id": "MyLlama-3b-v1-q4f32_0", "model_lib": "/url/to/myllama3b.wasm", } ], }; // override default const chatOpts = { "repetition_penalty": 1.01 }; // load a prebuilt model // pinch a chat action override and app config // nether the hood, it will load the exemplary from myLlamaUrl // and cache it successful the browser cache // The chat will besides load the exemplary room from "/url/to/myllama3b.wasm", // assuming that it is compatible to the exemplary successful myLlamaUrl. const engine = await CreateMLCEngine( "MyLlama-3b-v1-q4f32_0", { appConfig }, // engineConfig chatOpts, ); }

In galore cases, we only want to proviso the exemplary weight variant, but not needfully a caller exemplary (e.g. NeuralHermes-Mistral tin reuse Mistral's model library). For examples of really a exemplary room tin beryllium shared by different exemplary variants, see webllm.prebuiltAppConfig.

Build WebLLM Package From Source

NOTE: you don't request to build from root unless you would for illustration to modify the WebLLM package. To usage the npm, simply travel Get Started aliases immoderate of the examples instead.

To build from source, simply run:

npm install npm tally build

Then, to trial the effects of your codification alteration successful an example, wrong examples/get-started/package.json, alteration from "@mlc-ai/web-llm": "^0.2.84" to "@mlc-ai/web-llm": ../...

Then run:

cd examples/get-started npm install npm start

Note that sometimes you would request to move betwixt file:../.. and ../.. to trigger npm to admit caller changes. In the worst case, you tin run:

cd examples/get-started rm -rf node_modules dist package-lock.json .parcel-cache npm install npm start

In lawsuit you request to build TVMjs from source

WebLLM's runtime mostly depends connected TVMjs: https://github.com/apache/tvm/tree/main/web

While it is besides disposable arsenic an npm package: https://www.npmjs.com/package/@mlc-ai/web-runtime, you tin build it from root if needed by pursuing the steps below.

Install emscripten. It is an LLVM-based compiler that compiles C/C++ root codification to WebAssembly.
- Follow the installation instruction to instal the latest emsdk.
- Source emsdk_env.sh by root path/to/emsdk_env.sh, truthful that emcc is reachable from PATH and the bid emcc works.
We tin verify the successful installation by trying retired emcc terminal.

Note: We precocious recovered that utilizing the latest emcc type whitethorn tally into issues during runtime. Use ./emsdk instal 3.1.56 alternatively of ./emsdk instal latest for now arsenic a workaround. The correction whitethorn look like

Init error, LinkError: WebAssembly.instantiate(): Import #6 module="wasi_snapshot_preview1" function="proc_exit": usability import requires a callable
In ./package.json, alteration from "@mlc-ai/web-runtime": "0.18.0-dev2", to "@mlc-ai/web-runtime": "file:./tvm_home/web",.
Setup basal environment

Prepare each the basal limitations for web build:

In this step, if $TVM_SOURCE_DIR is not defined successful the environment, we will execute the pursuing statement to build tvmjs dependency:

git clone https://github.com/mlc-ai/relax 3rdparty/tvm-unity --recursive

This clones the existent HEAD of mlc-ai/relax. However, it whitethorn not ever beryllium the correct branch aliases perpetrate to clone. To build a circumstantial npm type from source, mention to the type bump PR, which states which branch (i.e. mlc-ai/relax aliases apache/tvm) and which perpetrate the existent WebLLM type depends on. For instance, type 0.2.52, according to its type bump PR #521, is built by checking retired the pursuing perpetrate https://github.com/apache/tvm/commit/e6476847753c80e054719ac47bc2091c888418b6 successful apache/tvm, alternatively than the HEAD of mlc-ai/relax.

Besides, --recursive is basal and important. Otherwise, you whitethorn brushwood errors for illustration fatal error: 'dlpack/dlpack.h' record not found.
Build WebLLM Package
Validate immoderate of the sub-packages

You tin past spell to the subfolders successful examples to validate immoderate of the sub-packages. We usage Parcelv2 for bundling. Although Parcel is not very bully astatine search genitor directory changes sometimes. When you make a alteration successful the WebLLM package, effort to edit the package.json of the subfolder and prevention it, which will trigger Parcel to rebuild.

Demo App: WebLLM Chat
If you want to tally LLM connected autochthonal runtime, cheque retired MLC-LLM
You mightiness besides beryllium willing successful Web Stable Diffusion.

This task is initiated by members from CMU Catalyst, UW SAMPL, SJTU, OctoML, and the MLC community. We would emotion to proceed processing and supporting the open-source ML community.

This task is only imaginable acknowledgment to the shoulders open-source ecosystems that we guidelines on. We want to convey the Apache TVM organization and developers of the TVM Unity effort. The open-source ML organization members made these models publically available. PyTorch and Hugging Face communities make these models accessible. We would for illustration to convey the teams down Vicuna, SentencePiece, LLaMA, and Alpaca. We besides would for illustration to convey the WebAssembly, Emscripten, and WebGPU communities. Finally, acknowledgment to Dawn and WebGPU developers.

If you find this task to beryllium useful, please cite:

@misc{ruan2026webllmhighperformanceinbrowserllm, title={WebLLM: A High-Performance In-Browser LLM Inference Engine}, author={Charlie F. Ruan and Yucheng Qin and Akaash R. Parthasarathy and Xun Zhou and Ruihang Lai and Hongyi Jin and Yixin Dong and Bohan Hou and Meng-Shiun Yu and Yiyan Zhai and Sudeep Agarwal and Hangrui Cao and Siyuan Feng and Tianqi Chen}, year={2026}, eprint={2412.15803}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/2412.15803}, }