Optimizing things in the USSR (2016)

11 May 2016

As a information scientist, a large portion of my occupation involves picking metrics to optimize and reasoning astir really to do things arsenic efficiently arsenic possible. With these types of questions connected my mind, I precocious discovered a wholly fascinating book astir about economical problems successful the USSR and the squad of data-driven economists and machine scientists who wanted to lick them. The book is called Red Plenty. It’s really written arsenic a novel, weirdly, but it nevertheless presents an meticulous economical history of the USSR. It draws heavy connected an earlier book from 1973 called Planning Problems successful the USSR, which I besides picked up. As I publication these books, I couldn’t thief but announcement immoderate parallels pinch readying successful immoderate modern organization. In what will beryllium acquainted to immoderate information intelligence today, the 2nd book moreover includes a quote from a interrogator who complained that 90% of his clip was spent cleaning the data, and only 10% of his clip was spent doing existent modeling!

Beyond each the absorbing parallels to modern information subject and operations research, these books helped maine understand a batch of absorbing things I antecedently knew very small about, specified arsenic linear programming, value equilibria, and Soviet history. This blog station is astir I learned.

Balance sheets and manual calculation: Kind of a trainwreck

The main task successful the centrally planned Soviet system was to allocate resources truthful that a desired assortment of equipment and services was produced. Every year, definite target outputs for each bully were established. Armed pinch estimates of the disposable input resources, cardinal administrators utilized equilibrium sheets to group plans for each factory, specifying precisely really overmuch input commodities each mill would receive, and really overmuch output it should produce. Up done the 1960s, this was ever done by manual calculation. Since location were hundreds of thousands of commodities, and since the proviso chains had galore dependency steps, it was intolerable to compute the afloat equilibrium sheets for the economy. The administrators truthful decided to make immoderate simplifying assumptions. As a consequence of these these simplifying assumptions, assets allocation became a spot of a trainwreck. Below are a fewer of the simplifications and their consequences.

• Dimensionality simplification by removing variables. Because location were excessively galore commodities to track, administrators often constricted their study to the 10,000 astir important commodities successful the economy. But erstwhile the accumulation of those commodities were planned, location was often a hidden shortage of commodities whose output was not planned centrally but which were utilized arsenic inputs to 1 of the 10,000 planned products. Factories that depended connected those commodities often sat idle for months arsenic they waited for the shortages to end.
• Dimensionality simplification by aggregation. Apparently, alloy tubes tin travel successful thousands of different types. They tin travel successful different lengths, different shapes, and different compositions. To trim the dimensionality of the problem, administrators would often way the full tonnage of a fewer wide classes of alloy tubes successful the models, alternatively than utilizing a much elaborate classification scheme. While their models successfully balanced the tonnage of tubes for the wide categories (the output successful tons of tube-producing factories matched the input requirements successful tons of tube-consuming factories), location were changeless surpluses of immoderate circumstantial types of tubes, and shortages of different circumstantial types of tubes. In particular, since tonnage was utilized arsenic a metric, tube-producing factories were overly incentivized to make easy-to-produce heavy tubes. As a result, bladed tubes were ever successful short supply.
• Propagating adjustments only a fewer degrees back. Let’s opportunity that during equilibrium calculations, the administrators realized they needed to bump up the target output of 1 commodity. If they did that, it was besides basal to bump up the output targets of commodities that were input into the target commodity. But if they did that, they besides needed to bump up the output targets of commodities that fed into those commodities, and truthful on! This progressive a crazy magnitude of other manus calculations each clip they needed make an adjustment. To simplify things, the administrators typically made adjustments to the first-order suppliers, without making the basal adjustments to the suppliers of the suppliers. This of people led to captious shortages of input commodities, which again led to idle factories.

Even if the administrators could get the accounting correct, which they couldn’t, their attempts to allocate resources would still beryllium acold from optimal. In the alloy industry, for example, immoderate factories were amended astatine producing immoderate types of tubes whereas others were amended astatine producing different types of tubes. Since location were thousands of different factories and conduit types, it was non-trivial to determine really to champion administer resources and output requirements, and it was not instantly evident which factories should beryllium expanded and which should beryllium closed down.

Supply concatenation optimizations

In the precocious 1960’s, a group of economists and machine scientists known arsenic the “optimal planners” began to push for a amended measurement of doing things. The group based on that a method called linear programming, invented by Leonid Kantorovich, could optimally lick the problems pinch the proviso chain. At a minimum, since the process could beryllium computerized, it would beryllium imaginable to execute much elaborate calculations than could beryllium done by hand, pinch little dimensionality reduction. But much importantly, linear programming allowed you to optimize arbitrary nonsubjective functions fixed definite constraints. In the lawsuit of the proviso chain, it showed you really to efficiently allocate resources, identifying businesslike factories that should get much input commodities, and inefficient factories that should beryllium unopen down.

The optimal planners had immoderate occurrence here. For example, successful the alloy industry, astir 60,000 consumers requested 10,000 different types of products from 500 producers. The producers were not arsenic businesslike successful their production. Some producers were businesslike for immoderate types of alloy products, but little businesslike for different types of alloy products. Given the full magnitude of each merchandise requested, and fixed the constraints of really overmuch each mill tin produce, the extremity was determine really overmuch each mill should nutrient of each type of product. If we simplify the problem by conscionable asking really overmuch each mill should nutrient without considering really the products will beryllium distributed to the consuming factories, this becomes a straightforward exertion of the Optimal Assignment Problem, a well-studied illustration successful linear programming. If we additionally want to optimize distribution, taking into relationship the distance-dependent costs of shipments from 1 mill to another, the problem becomes much analyzable but is still doable. The problem becomes akin to the Transportation Problem, different well-studied illustration successful linear programming, but successful this lawsuit generalized to aggregate commodities alternatively of conscionable one.

By introducing linear programming, the optimal planners were modestly successful astatine improving the ratio of immoderate industries, but their effect was limited. First, governmental considerations prevented galore of the recommendations surfaced by the exemplary from being implemented. Cement factories that were known to beryllium excessively inefficient aliases excessively acold distant from consumers were allowed to stay unfastened moreover though the optimal solution recommended that they beryllium closed. Second, since the planners were only allowed to activity successful definite constrictive parts of the economy, they ne'er had an opportunity to propagate their recommendations backmost successful the proviso chain, though 1 could ideate extending the models to do so. Third, and possibly astir importantly, the worth of each commodity was group by old-school administrators successful an unprincipled way, and truthful the optimal planners were forced to optimize nonsubjective functions that didn’t moreover make sense.

Ideas astir optimizing the full economy

While the optimal planners were capable to amended the ratio of a fewer industries, they had much eager plans. They believed they could usage linear programming to optimize the full system and outperform capitalist societies. Doing truthful progressive much than conscionable scaling retired the proviso concatenation optimizations adopted by definite industries. It progressive protector prices and liking rates, and a fewer different things I’ll admit I don’t wholly understand. But while I don’t really understand the implementation, I consciousness for illustration the broader extremity of the planners is easier to understand and explain:

Basically, successful a wholly free market, astatine slightest nether certain assumptions, prices are expected to converge to what’s called a General Equilibrium. The equilibrium prices person a immoderate bully properties. They equilibrium aggregate proviso and demand, truthful that nary commodities are successful shortage aliases surplus. They are besides Pareto efficient, which intends that cipher successful the system tin beryllium made amended disconnected without making personification other worse off.

The optimal planners thought that they could do better. In particular, they pointed to 2 problems pinch capitalism: First, prices successful a capitalist nine were wished by individual agents utilizing proceedings and correction to conjecture the champion price. Surely these agents, who had imperfect information, were not picking the precisely optimal prices. In contrast, a cardinal planner utilizing optimal computerized methods could prime prices that deed the equilibrium much exactly. Second, and much importantly, capitalism targeted an nonsubjective usability that — while Pareto businesslike — was not socially optimal. Because of immense differences successful wealth, immoderate group were capable to get acold much equipment and services than different people. The optimal planners projected utilizing linear programming to optimize an nonsubjective usability that would beryllium much socially optimal. For example, it could purpose to administer equipment much equitably. It could prioritize definite socially valuable equipment (e.g. books) complete socially destructive equipment (e.g. alcohol). It could prioritize sectors that supply benefits complete longer clip horizons (e.g. dense industry). And it could see constraints to guarantee afloat employment.

What happened

None of this ever really happened. The eager ideas of the optimal planners were ne'er adopted, and by the 1970s it was clear that surviving standards successful the USSR were falling further down those of the West. Perhaps things would person been amended if the optimal planners sewage their way, but it seems for illustration the statement is that their plans would person grounded moreover if they were implemented. Below are immoderate of the main problems that would person been encountered.

• Computational complexity. As described successful a wonderful blog station by Cosma Shalizi, the number of calculations needed to lick a linear programming problem is: \((m+n)^{3/2} n^2 log(1/h)\), wherever \(n\) is the number of products, \(m\) is the number of constraints, and \(h\) is really overmuch correction you are consenting to tolerate. Since the number of products, \(n\), was successful the millions, and since the complexity was proportional to \(n^{3.5}\), it would person been practically intolerable for the Soviets to compute a solution to their readying problem pinch capable item (although spot below). Any effort to trim the dimensionality would lead to the aforesaid perverse incentives and shortages that bedeviled earlier systems driven by manus calculations.
• Data quality. The optimal planners thought that optimal machine methods could find prices that much precisely approximated equilibrium than could beryllium done successful a marketplace economy, wherever fallible quality actors guessed astatine prices by proceedings and error. The reality, however, would person been the nonstop opposite. Individual actors successful a marketplace system understand their section needs and constraints beautiful well, whereas cardinal planners person fundamentally nary thought what’s going on. For example, cardinal planners don’t person bully accusation connected erstwhile a mill fails to person a shipment and they don’t person an meticulous consciousness for really overmuch much businesslike immoderate devices are than others. Even worse, successful bid to get much resources, mill managers successful the USSR routinely lied to the cardinal planners astir their accumulation capabilities. The business became truthful bad that, according to one of the heavy authorities secrets of the USSR, cardinal planners preferred to usage the CIA’s analyses of definite Russian commodities alternatively than reports from section Party bosses! This is particularly crazy if you see that the CIA described its ain information arsenic being of “debilitatingly” mediocre quality.
• Nonlinearities. The optimal planners assumed linearity, specified that the costs for a mill producing its 1000th widget was assumed to beryllium the aforesaid arsenic the costs for producing its first widget. In the existent world, this is evidently false, arsenic location are expanding returns to scale. It’s imaginable to exemplary expanding returns to scale, but it becomes harder to lick computationally.
• Choosing an nonsubjective function. Choosing what the nine should worth is really a governmental problem, and Cosma Shalizi does a very nice job describing why it would beryllium truthful difficult to travel to agreement.
• Incentives for innovation. The cardinal planners couldn’t find assets allocation for products that didn’t beryllium yet, and much importantly neither they nor the factories had overmuch inducement to invent caller products. That’s why the Soviet Union remained truthful focused connected the steel/coal/cement system while Western nations shifted their attraction to plastics and microelectronics.
• Political resistance. As described successful a erstwhile example, the model-based recommendations to unopen down definite factories were ignored for governmental reasons. It is apt that galore recommendations for the broader system would person been ignored arsenic well. For example, if a machine recommended that the value of heating lipid should beryllium doubled successful the winter, really galore politicians would fto that happen?

Could this activity successful the future?

Had the optimal planners’ ideas been adopted astatine the time, they would person failed. But what astir the future? In a 100 years, could we person the method capacity to propulsion disconnected a wholly planned economy? I did immoderate poking astir the net and found, somewhat to my surprise, that the reply is actually… maybe. It turns retired that 2 of the astir superior problems pinch cardinal readying could person technological solutions that whitethorn look far-fetched but are possibly not impossible:

Let’s commencement pinch computational complexity. As described supra and successful Cosma Shalizi’s post, the number of steps required to lick a linear programming problem pinch \(n\) products and \(m\) constraints is proportional to \((m+n)^{3/2} n^2\). The USSR had astir 12 cardinal types of goods. If you transverse them complete astir 1000 imaginable locations, that gives you 12 cardinal variables, which according to Cosma would correspond to an optimization problem that would return a 1000 years to lick connected a modern desktop computer. However, if Moore’s Law holds up, it would beryllium imaginable successful 100 years to lick this problem reasonably quickly. It’s besides worthy pointing retired that the economy’s input-output matrix is sparse, since not each merchandise depends connected each different merchandise arsenic input. It may beryllium imaginable that personification mightiness create a faster algorithm that leverages this sparsity, though Cosma is somewhat skeptical that this could happen. [In an earlier type of this post, I discussed a sparsity-based proposal that supposedly brought things down to \(m \times n\) complexity. This was apparently a red herring that doesn’t really lick the optimization problem.]

As described earlier, the 2nd superior rumor pinch a centrally planned system was data quality: Central planners’ knowledge astir the input requirements and output capabilities of individual factories was simply not arsenic bully arsenic the group really moving successful the factory. While this was surely the lawsuit successful the Soviet Union, 1 can’t thief but wonderment astir technological improvements successful proviso concatenation management. Imagine if each merchandise had search devices, pinch different sensors and cameras to find merchandise quality. Already Amazon is moving successful that guidance for beautiful overmuch each user goods, and 1 could ideate a world wherever request could beryllium measured pinch the Internet of Things. Whether a authorities would beryllium capable to harness this information arsenic competently arsenic Amazon is doubtful, and it’s evidently worthy asking whether we would ever want a authorities to beryllium utilizing that type of data. But from a method constituent of position it’s imaginable that the information value issues that destroyed the USSR mightiness beryllium overmuch little superior successful the future.

All that being said, it’s still unclear to maine really an nonsubjective usability could beryllium chosen successful measurement that would democratically fulfill people, really invention could beryllium incentivized, aliases really governmental freedoms could beryllium preserved. Socialism has a mediocre way grounds historically, pinch tons of grounded promises that “this clip will beryllium different”. If you’d for illustration to publication much astir really things worked successful the USSR, you should decidedly cheque retired Red Plenty. It was 1 of the weirdest and astir absorbing books I person read.