TLDR;
I implemented a chart map-reduce utilizing Apache DataFusion. Where possible, I offloaded everything to disk, and designed the algorithms to trust connected bulk scans alternatively than random access. DataFusion handles spillover, sort-merge joins, aggregations, readying and execution, truthful my codification is very lightweight. I tested it successful strict mode by moving it via systemd-run pinch a difficult representation limit. It works. Of course, I person encountered immoderate issues: for example, I often acquisition deadlocks from FairSpillPool successful utmost scenarios, and I person not yet recovered a measurement to make SMJ usage pre-sorting of the information connected disk. But it works. I tin compute PageRank connected a directed chart pinch 1 cardinal edges (graph500-26 from the Graphalytics dataset) utilizing 5 GB of memory. Alternatively, I tin place each the weakly connected components successful a chart pinch 2 cardinal edges (twitter_mpi from the aforesaid dataset collection) utilizing 10 GB of memory. Neither NetworkX nor Igraph tin do this; astir existing chart algorithms require the chart to fresh into memory. Previously, I thought you needed Apache Spark and GraphFrames for billion-scale chart analytics. Now, however, I deliberation each you request is simply a laptop. I person wholly changed my aged opinion astir utilizing Apache DataFusion for chart analytics.
Setup
I tested 2 tasks.
PageRank
What is PageRank?
The task is to compute PageRank connected graph500-26 from Graphalytics dataset:
| Num nodes | 32,804,978 |
| Num edges | 1,051,922,853 |
| Directed | False |
| Memory Limit | 5 GB |
| DataFusion Pool Size | 4 GB |
PageRank is 1 of the astir celebrated chart centrality algorithm and is utilized from hunt results ranking to anti-fraud scoring. My DataFusion implementation is classical Pregel: bulk-synchronous parallel algorithm (aka Map-Reduce) which I expressed utilizing joins and aggregate. Very akin to what is successful the halfway of Spark's GraphFrames library.
Weakly Connected Components
What are Weakly Components?
The task is to place each the weakly connected components connected twitter_mpi from the aforesaid dataset:
| Num nodes | 52,579,682 |
| Num edges | 1,963,263,821 |
| Directed | True |
| Memory Limit | 10 GB |
| DataFusion Pool Size | 8 GB |
WCC is the halfway portion of immoderate personality (entity) solution problem. For example, erstwhile you request to do information deducplication from different strategy done transitive IDs you extremity up pinch WCC problem. My DataFusion implementation is based connected the "In-database connected constituent analysis", Bögeholz et al., arXiv 1802.09478. I already implemented the aforesaid algorithm for the Spark's GraphFrames truthful it was an evident choice.
Results
PageRank
An easy part. I utilized SMJ conscionable to proove the scalability but it is besides imaginable to usage HJ because vertices are mini (32M) and PageRank authorities is trivial: 1 file rank (f64), 1 file out-degree (i64), connected information emblem (bool). With HJ it is faster. PageRank useful complete directed edges truthful it deos not require to symmetrize the graph. Just offload edges to disk and iterate by updating authorities (and offload to disk arsenic good to break the lineage) until converged.
The compute clip is long: astir 30 minutes for 15 afloat iterations. But the setup is astir memory, not speed. Give it immoderate much realistic numbers for 1B chart analytics and it will activity accelerated capable (I tested). I checked numbers against the crushed truth: 100% lucifer (with 0.0001 tolerance). A batch of optimizations tin beryllium done present arsenic well: successful mentation it is imaginable to bucket edges by scope aliases do benignant of range-partitioing, truthful the SMJ does not request to benignant again the biggest join-side (edges) connected each loop to get triplets. As good I'm not 100% parquet is the champion prime here. Also would beryllium absorbing to effort to fuse join+agg: each Pregel iterarion is for illustration edges <-[join] nodes-state -> group by + agg -> [join] -> nodes-state -> update nodes-state. If I tin fuse together the first 2 stages it tin beryllium a immense triumph from the capacity constituent of view. Meanwhile I do not cognize yet really to do it successful DataFusion: a batch of point to learn.
WCC
The hardest part. 2B edges twitter chart is already immense (its edges are 30 GB successful CSV !!!). But for WCC we request to symmetrize edges (or do a national betwixt src, dst and dst AS src, src AS dst + chopped connected top), truthful astatine the highest we are crunching almost 4 cardinal of edges utilizing only 8GB DataFusion pool. After the travel survives the first fewer iteration, contraction process trim the magnitude of edges dramtically and algorithm ends successful 10 minutes pinch debased representation pressure.
Results are correct: Graphalytics provides crushed truth and it is easy to check:
English (US) · 
Indonesian (ID) · 
