Neki

Neki is now disposable successful level preview.

Neki is built from lessons we’ve learned complete 8 years of moving immoderate of the largest sharded MySQL clusters successful the world. Thousands of accumulation workloads pinch millions of queries per 2nd for companies wherever moreover a fewer seconds of downtime is simply a very nationalist event. We cognize what it intends to powerfulness the world’s biggest tier 0 workloads.

When we released PlanetScale Postgres a twelvemonth and a half ago, we knew we needed to do more. In that clip we’ve onboarded respective thousands of customers connected PlanetScale, immoderate of them rivaling the size of our largest MySQL customers. Time and clip again, we watched teams attack the ceiling of a azygous instrumentality pinch Postgres. Metal bought them time, but pinch customers hitting the precocious limit of what a azygous instrumentality is tin of, we recovered location was nary bully action to manus them. Enter Neki.

Why Neki?

You already cognize the problems that travel pinch fast-growing Postgres databases: tables excessively ample to vacuum aliases scale without affecting traffic, backups taking hours, relationship limits, attraction windows for schema changes, transaction wraparound and truthful overmuch more.

You tin move to a bigger instance, but yet you tally retired of large capable machines, and the problems don’t standard linearly arsenic you adhd much cores and IOPS.

The existing answers each inquire you to springiness thing up. Application-level sharding pushes routing into your code. Postgres-”compatible” distributed databases hide the shard cardinal from you, return distant your extensions, and adhd complexity and latency which becomes difficult to grip and debug.

So we built Neki pinch a fewer principles, the biggest 1 being: instrumentality to Postgres, don’t activity astir it, clone it, aliases move distant from it.

How does Neki work?

We architected Neki from first principles for Postgres, pinch existent Postgres connected each shard. There are 4 moving parts.

Neki routers

Your exertion first connects to a Neki router. The router speaks the Postgres ligament protocol truthful your existing drivers and ORMs support moving pinch a azygous relationship string. A router has a afloat Postgres query parser, a distributed query planner, query buffering and more. It parses your query, builds a scheme that decides which shards should tally it, sends the activity out, and combines the results backmost into 1 stream. Routers tin standard vertically and horizontally, truthful nary azygous router becomes the bottleneck.

Sharding and shard groups

Every shard successful Neki is existent Postgres pinch 1 superior and astatine slightest 2 replicas, dispersed crossed readiness zones. There is nary modified retention engine. Extensions, SQL support, and capacity behave the measurement Postgres behaves, because it is Postgres.

Shards are organized into shard groups, truthful different tables aliases workloads tin unrecorded connected different sets of shards. Each shard uses a configuration floor plan that defines its lawsuit size, replica count, storage, Postgres parameters, and extensions, truthful you tin size each group for its ain traffic.

Connection pooling

Sidecars tally alongside each Postgres instance. This is the portion that makes Neki's relationship handling meaningfully amended than conscionable sticking PgBouncer successful beforehand of a database. Because Neki controls some ends of the connection, the router broadside and the Postgres side, it tin size pools to what each lawsuit tin really service alternatively of estimating from extracurricular the process.

Control plane

The power level tracks the wellness of each node, runs planned switchovers and unplanned failovers, and coordinates the workflows that reshard data, use schema changes, and execute type upgrades.

Data topology

Tying it together is the data topology, a JSON configuration that maps your logical tables onto beingness shards. You specify shard indexes, which specify the file Neki routes connected and really that worth gets hashed, and shard groups, which power really galore shards a group of tables spreads crossed and which shards those are. Routers cache the topology and consult it connected each plan.

What you get beyond sharding

Everything you would usually schedule a attraction model for runs arsenic a built-in workflow successful Neki. Workflows proviso caller target nodes, drawback them up pinch replication, move postulation pinch a __neki metafunction, and discontinue the aged nodes. All done the aforesaid psql relationship your exertion uses.

This online operations exemplary covers schema changes, type upgrades, planned and unplanned failovers, imports, and resharding.

Neki besides includes each of the features you’ve travel to trust connected pinch PlanetScale: Insights, schema recommendations, branching, MCP, and more.

You tin besides tally Neki unsharded, arsenic a azygous superior pinch replicas. You get the improved relationship pooling, online DDL, zero downtime upgrades, and wellness monitoring earlier you request to shard. When you do, resharding is simply a workflow you tally against the cluster you already have.

What is simply a level preview?

We wanted to get Neki into your hands arsenic soon arsenic possible. You should not tally accumulation workloads connected Neki during the level preview. The merchandise is still changing, and immoderate of those changes will beryllium breaking.

If you person immoderate feedback, questions, aliases look immoderate issues during the level preview, please fto america know. Fill retired a support ticket aliases join our Discord

Try Neki today

Sign successful to PlanetScale, opt successful to the level preview, and create a Neki cluster. Read the Neki docs for much accusation astir Neki's architecture, really to shard, and more.

If you person a ample Postgres cluster and are funny whether Neki is simply a bully fit, get successful touch. We would emotion to do a backstage demo for your team, excavation into your schema and query patterns, and springiness you existent suggestions connected really to shard.