Brussels / 1 & 2 February 2025

schedule

Huge graph analysis on your own server with WebGraph in Rust


Huge graphs, from billions to trillions of nodes and arcs, are more and more common in modern data-intensive applications. Graph compression techniques allow loading and manipulating such graphs into main memory of either high-end workstations or commercial-grade servers, depending on graph size. In this talk, we report about webgraph-rs, a recent clean-slate Rust re-implementation of WebGraph, a state-of-the-art graph compression framework, formerly implemented in Java. webgraph-rs comes as a series of interconnected Rust crates of general interest, including high-performance bit streams, zero-copy (de)serialization, constant-time measurement of heap size for large data structures, and high-performance implementation of succinct data structures . Using webgraph-rs one can load graphs such as the Software Heritage Merkle DAG (about 0.6 trillion edges ) in less than 200 GiB of RAM and visit it in a few hours with a single core.

Speakers

Photo of Sebastiano Vigna Sebastiano Vigna