Cloudflare operates at a measure so big that equal following operating current for years, it doesn’t appear real. We have thousands of servers all complete the earth alongside petabytes of RAM and millions of CPU cores, and all of it is pushed to the max. As huge as those resources feel, they are motionless finite, and whenever you need all assistance to run on all node, it doesn’t depart area for wasted space.
At this scale, small improvements are greatly magnified, so equal 1%-at-a-time improvements are value celebrating. And several tweaks add up to a lot more: in this post, we’ll appearance at how small changes to a sole algorithm reduced the recollection footprint of one of our Pingora-based services significantly. That allowed us to reclaim additional than 100TB of RAM globally, on top of the 100TB of recollection the DNS squad was capable to shed final month.
Waste not
Maintaining equitable asset sharing between teams is not easy, particularly in ample organizations. One of the ways Cloudflare ensures the balance is kept is through the tireless efforts of the fantastic Performance team.
This narrative starts alongside a ticket submitted by Ivan who found: Excessive recollection use from pingora-ketama in Pingora Backend Router. The finding was that our inner load-balancing service, Pingora Backend Router (yes, PBR), was using considerably additional recollection than expected — specifically in structures connected alongside pingora-ketama, which is our open-source archive for handling accordant hashing.
In command to conversation concerning how we addressed this seeming overuse of memory, we need to conversation concerning what accordant hashing equal is, why we are using it in PBR, and how it became so recollection hungry. Along the way, we’ll study several Rust and equal a small math.
Consistent hashing
Consistent hashing is a extensively used method for distributing tasks throughout multiple servers in a way that does not necessitate ample changes whenever servers are added or removed. Internally we use it to path cacheable requests to servers by URL. This allows us to keep lone one copy of a document stored per data center and gives a stable way to discover the location of all file. We have mentioned this system before, but let’s obtain the period to stroll through how and why this algorithm is used and how it works.
The key idea of accordant hashing is that during hash functions can obtain any benevolent of input, their output is constricted to a sole unsigned entire figure (32, 64, or 128-bit integers depending on which hash function). This allows us to connect tasks and servers to all another in a consistent way. Most discussions of accordant hashing have you think of that output area as a continuous, circular circle that wraps about from its max value to zero. This depiction makes for several nice visualizations, but it can additionally create the uncomplicated idea of entire figure ranges appear additional complex than it needs to be. For our discussion, we’ll portray the 32-bit output of our hash function as a figure line.
Now, let’s say we have a set of servers, A, B, & C, and a set of tasks t-z. We can map all onto the figure row according to the hash of their delegate values, so item akin IP addresses for servers and cache keys for tasks.
Assigning tasks to servers is now fair a matter of finding the archetypal server to the remaining of all task. We can portray this visually by coloring in the area of hashes that volition be connected alongside all server. Notice that the range covered by server C wraps about to the beginning, hence the idea that hashes be in a ring.
And that’s it. At a basis level, accordant hashing is this uncomplicated — but it doesn’t obtain lengthy to see that there is area for improvement. Notice that the range covered by server A in our example is considerably larger than that of either B or C. This is a issue since the fraction of the requests a server handles is going to be proportional to the size of its range on the figure line. Ideally we would akin to justify all server volition have an equal size, but since hashes are basically random numbers, we have to conversation concerning the size of the regions in conditions of statistics. 😨
Math and consequences
First: don’t panic. I commitment I'm not concerning to lie to you and that we volition remain safely inside the limits of a day-one probability lesson. When we conversation concerning statistical distributions, there are two big factors that assistance us quantify doubt in helpful ways: expected value and standard deviation. In (over-)simplified terms, expected value gives us a item anywhere measurements according to a allocation volition be centered, and norm deviation tells how near to that chief item most measurements are apt to be.
For accordant hashing, we can compute these factors for the partial size of the range connected alongside one of N servers. (Details on anywhere this equation comes from later).
$$m \begin{align*} \text{Exp} &= \frac{1}{N} \\ \text{SD} &= \frac{1}{N}\sqrt{\frac{N-1}{N+1}} \end{align*} m$$
In conditions of tangible numbers, let’s say we have 100 servers. The formulas complete give:
$$m \text{Exp}=1/100 = 1\% \\ \text{SD}= \frac{1}{100}\sqrt{\frac{100-1}{100+1}} \approx 0.99\% m$$
That tells us that we can anticipate that the range all server handles volition be centered about 0.99% of the total and most of the lengths to autumn inside 1% of what's expected. This sounds fine until we acknowledge that that’s 0.99% of the total length. We need to measure the norm deviation by the expected value to see how big the error is as a fraction of the mark size. This value is called the coefficient of variation.
$$m \text{CV} = \frac{\text{SD}}{\text{Exp}} = \sqrt{\frac{N-1}{N+1}} m$$
At $m N=100, \text{CV} \approx 99\% m$ — definition several servers volition apt be operating 99% harder than they have to be (handling twice as many requests) during others could be doing practically nothing! Now that we have a way to foretell how evenly loaded servers volition be using accordant hashing, we can commencement operating on improvements.
What if we add hashes?
The simplicity of accordant hashing is a double-edged sword. It’s uncomplicated to comprehend and execute since everything is turned into easily-relatable hashes on the identical numberline, but any improvements to the scheme volition additionally need to be relatable to that numberline. That method the resolution to any accordant hashing issue can lone be more hashes. It’s small akin a golden hammer (a tool alongside which all problems appearance akin nails) and additional akin a aureate nail in that it turns all tools into hammers.
To resolve the issue of imbalanced workloads, we can add multiple hashes to portray all server alternatively of fair one. We’ll get to the math rearward this momentarily, but it should create several intuitive awareness that during all idiosyncratic range has a ample norm deviation, adding a bunch together should create their total size equal out. If we obtain our three-server example from the complete diagrams and add two additional hashes at random for all server, we see that it helps equal out all server’s workload.
This is an admittedly contrived example. The random nature of the scheme method there’s no justify how much betterment you volition get from adding 2 additional hashes per server, but it should create several intuitive awareness that combining additional of these hash segments together produces a additional equal distribution. Each section in the sum has a chance of balancing another. Maybe one is too short; perchance one is too long. This is basically what the law of ample numbers tells us should happen… The apparent issue is it lone plant for ample numbers. In NGINX, the baseline figure of hashes per server is hardcoded to 160, and Pingora uses the same value as the default. I’ll spare you the math for now, but if we go rear to our 100-server example, if we use 160 points per server alternatively of fair one, the coefficient of assortment (which we can think of akin an error margin) drops from concerning 99% to concerning 8%, a important improvement.
What if we add additional hashes?
We saw complete that expanding the figure of hashes per server by a changeless amount allows us to enhance how evenly workloads are distributed per server, but what if we don’t desire to allocate the activity evenly? In Cloudflare’s case, we have several servers that have additional retention area than others, so it would be improved to have the figure of requests allotted to a server be proportional to its disk space. One way to accomplish this is alongside the ketama algorithm. The naming is a small comic since the algorithm is named following the library anywhere it was archetypal implemented, and the archive was named … fine you can google it 😶🌫️.
The entire algorithm boils downward to: For any two servers, $m S_1m$ & $mS_2m$, if we desire the requests served by $mS_1m$ to be $mw\timesm$ additional than those served by $mS_2m$, the figure of hashes connected alongside $mS_1m$ needs to be $mH_1 = w\times H_2m$. This allows us to set a “weight” for all server, which scales the figure of hashes connected alongside that server. Unfortunately this is not a replacement for the changeless measure aspect we added in the division above. That scaling needs to be there to set a minimum error margin, which volition display up in the servers alongside the lowest weights.
For us, since we desire workload to be scaled according to storage, we can use the disk area as the weight, which is exactly what the Pingora squad has been doing for years. Elsewhere in the business anywhere workloads are additional compute-intensive, weights power be according to CPU or GPU count.
What if we add equal more hashes???
The final issue we need to location is that so far we are operating under the assumption that any server can grip any request, but in custom that is not the case. Things akin compliance requirements or enabled caching features average lone a subset of servers can grip any particular request. Unfortunately, dissimilar before, we can’t resolve this issue by adding additional hashes to the identical ring. We have to add completely new rings, and not lone that — all combination of features possibly needs its own particular ring!
Duplication according to combinations is a traditional formula for exponential explosion. In our case, we have a fistful of distinct features foremost to $m2^\text{handful} = \text{dozens}m$ of distinct accordant hash rings. So as you have likely guessed by now, the "excessive recollection use" (6GB in several cases) that Ivan established was because of an enormous figure of hashes to modify all the functionality we need and which have to be stored in memory. So what can we do?
Storage improvements
One big betterment came from Zaidoon, who had an understanding concerning our struct for storing hashes in PBR. That struct looks akin this:
struct Point { hash: u32, index: u32, }
In recollection this is represented as eight bytes, anywhere four go to the hash (which is unavoidable), and four go to an indicator pointing to the server which is stored in another array. Zaidoon’s understanding was that a 32-bit entire figure for that indicator is wasteful, since PBR is not apt to always have to coordinate additional than $m2^16 \approx 65\text{k} m$ servers at the identical time, so a 16-bit entire figure volition work. So we can substitute the struct complete alongside this one:
struct PointV2 { hash: u32, index: u16, }
Unfortunately, Rust doesn’t create it that easy. Changing the size of the indicator as we did complete does nothing to decrease the recollection footprint. This is since Rust has alignment rules that necessitate the size of a construction in recollection to be a multiple of its largest (or “most aligned”) field. In this case, the hash is the largest alongside four bytes, so whenever stored in memory, a Point is required to have size $mN \times 4m$, so the minimum size is eight bytes.
Luckily there are well-known ways about this. You (meaning me) power be tempted to use #[repr(packed)], but that is arguable for good reasons. A safer but small readable resolution is to shop the hash and indicator as raw byte gathering and admission them alongside getters. Both methods compile to the identical thing.
struct Point([u8; 6]); impl Point { fn hash(&self) -> u32 { u32::from_ne_bytes(self.0[0..4].try_into().unwrap()) } fn index(&self) -> u16 { u16::from_ne_bytes(self.0[4..6].try_into().unwrap()) } }
This uncomplicated (if wordy) alter reduces the amount of recollection used for accordant hashing by a whopping 25%! In command to do improved than that, we’ll need to jump rear into the math, so everybody suspend on to something; this is the residence stretch.
What if we tried small hashes?
You may have noticed that we gave the equation for the norm deviation for the case anywhere there is lone one hash per server. Deriving the equation for the case anywhere there are $m k m$ hashes per server is not easy, and most sources lone provision you an approximation or an asymptotic limit, but not us. I power not be a statistician, but I grew up alongside a calculus instructor (Hi, Mom!), and I wanted to cognize the actual value. The complete derivation is in a supplemental post, but current is the payoff.
$$m \text{Exp}_k = \frac{1}{N}, \text{SD}_k=\sqrt{\frac{(k+1)}{N(kN+1)}-\frac{1}{N^2}} m$$
To see how expanding the hash figure improves the accuracy, we need to appearance again at the coefficient of variation.
$$m \text{CV}_k=\frac{\text{SD}_k}{\text{Exp}_k}=\sqrt{\frac{N-1}{(N*k+1)}} m$$
Plotting $m\text{CV}_km$ shows a possible issue alongside the “just add additional hashes” mentality (other than overusing RAM).
You can see all stage downward in error border requires (almost) an command of dimension addition in the figure of hashes per server, so adding additional hashes yields small and small improvement. Recall that we are using a basis of 160 hashes scaled by the server's retention size. To create the math easier, we'll say the weighting aspect $m{m_w}m$ for a server is 625, so we get $m{k = 160\times625 = 100{,}000}m$. We can see from the diagram complete that the final 90,000 hashes we added are purchasing us a minuscule 0.7% decrease in error. Unfortunately things get equal worse from there.
The predictions from my beautiful math lone activity if we think concerning hashes in a continuous ring, but in custom we use 32-bit numbers for the hashes that have the possible for collisions, and the probability of collisions goes up amazingly quickly as the figure of hashes increases (see the birthday paradox). Collisions matter since in the ideal case, all hash contributes to the quantity and allocation of requests handled by the connected server, but a collision method several contributions are randomly dropped, introducing unpredictable error. If we difference several simulated results alongside 32-bit hashes alongside the predicted error rate, we can see that for data centers alongside 2048 servers, the error charge increases: between 10,000 and 100,000 hashes per server.
Ultimately, equal although this realization feels benevolent of bad, it’s awesome news for our scheme to reclaim several RAM! Now that we have several math to rear it up, we resolute that we could decrease the figure of hashes we were generating for all server by 90% without incurring any appreciable error, so that is what we set out to do.
Migrating without liquefying origins
There was one additional problem: changing the hash circle changes anywhere several cacheable requests go. Even if the new circle is better, switching the entire network at formerly would efficiently invalidate nearly all cached content. It would rotate a recollection optimization into an apocalyptic addition in base traffic.
So we did not create this a sole earth flip. For a while, PBR carried the two versions of the cacheable burden balancer in memory: the old ketama circle and the new smaller one. Each petition used our normal immigration example to decide which circle should choose the backend. That meant the rollout decision was stable per petition hash, and it additionally gave us a spotless rollback path. If item looked wrong, we could dispatch new requests rear through the old circle without redeploying PBR.
We afterward rolled the immigration out in layers. We started alongside small validation locations, moved through increasingly larger groups of data centers, and lone afterward continued toward the remainder of the world.
The crucial part was that we controlled two dimensions independently: how much traffic used the new ring, and anywhere that traffic was allowed to move. A plain earth percent rollout would have dispersed cache churn everyplace at once. Data-center-scoped rollout kept the blast radius small and made it much easier to inform whether a alter was really safe.
During the migration, we watched backend-selection traces, ring-version counters, PBR association errors, procedure memory, startup time, cache behavior, and base traffic. Once the immigration reached 100%, we removed the impermanent old-ring path, and voila!
The diagram complete shows the difference of the recollection used by PBR the week of the alter compared alongside data from a few weeks before, as fine as the outcome of subtracting one from the other. The keen autumn is the day anywhere the type of PBR alongside the ample (now unused) hash rings was decommissioned forever. Looking at the difference, we get the satisfying outcome that our changes dropped the used recollection by 100TB!
Try it yourself
All the changes we talked concerning in this article are accessible now in the pingora-ketama crate in the form of a (for now) unadvertised cargo feature. The v2 circle has the compacted retention format, a faster sorting method, and the capability to measure the basis figure of hashes per node. Our concentration in making these changes had to be on stability and control, so the v1 circle is identical to what pingora ketama has continually used, and the archive makes it imaginable to run the two simultaneously and decide on a request-by-request basis which to use and when.
Beyond trying our literal accordant hashing changes, I would akin you to obtain distant from this several inspiration to dig into your own systems to see what “simple” or “obvious” decisions are hiding possible wins, if you’re consenting to enter the numbers. You power not be capable to resolve all your problems alongside Rust, but math is universal.
