27.5KB language-agnostic WebGPU syntax highlighter

Sep 09, 2026 08:14 AM - 5 days ago 5

Shu Ding astatine Vercel Labs

import { item } from 'gpu-lexer' const spans = await highlight('source code') // { // type: 'plain' | 'comment' | 'string' | 'number' | 'keyword' | 'type' | 'function' | 'constant' | 'operator' // start: number // end: number // }[]

gpu-lexer splits root codification into elemental parts—words, whitespace, newlines, and symbols. Then a mini WebGPU exemplary combines section and whole-file discourse to explanation each part. It is designed for any language: alternatively of choosing a grammar, it guesses each part's type from the surrounding source, moreover erstwhile it ne'er saw that connection aliases syntax during training. Adjacent labels go the item spans returned to your code.

This is an experiment, not a grammar-equivalent highlighter. On files kept retired of training, 12.57% of the existent model's token labels disagree from Shiki. This measures statement pinch Shiki—not nonsubjective correctness—and unseen languages aliases real-world codification whitethorn disagree much often.

Highlight 10× three.min.jswarmed browser clip · little is better

One browser tally aft 1 warm-up connected September 8, 2026. The input was 10 concatenated copies of three.min.js (5.56M characters). MacBook Pro, Apple M4 Pro, 20-core GPU, 24GB, macOS 26.6.2, Chrome 152. Each motor ran successful a dedicated worker; DOM rendering was excluded. gpu-lexer and Shiki returned token data, Starry Night returned a HAST tree, while Sugar High, Prism.js, and Highlight.js returned highlighted HTML. Sugar High 2.3.1, Prism.js 1.30.0, Highlight.js 11.12.0, Starry Night 3.11.0, and Shiki 4.4.3.

Loaded room sizeruntime + selected connection sum · little is better

Minified and Brotli-compressed browser bundles measured connected September 8, 2026. Major web includes javascript, typescript, css, html, json, and markdown. gpu-lexer uses the aforesaid bundle for each language. Starry Night totals see its Oniguruma WASM payload.

Top-25 weighted correctnesspopularity-weighted statement pinch Shiki · higher is better

Shiki is the 100% normalization reference. Each library's token names are mapped to the aforesaid 9 classes: plain, comment, string, number, keyword, type, function, constant, and operator. Scores comparison non-whitespace root parts crossed 1,069 held-out files successful the GitHub Innovation Graph apical 25 for 2026-Q1, weighted by each language's pusher count. Unsupported languages people zero; corpus size does not impact the weights.

[ unrecorded demos ]

75 languages

react.development.js0 bytes