Hi all, I would like to propose a new direction for OLAP in TinkerPop. The proposal is to remove hadoop-gremlin in 4.x, deprecate spark-gremlin in 4.x, retain VertexProgram for BSP-style processing, and eventually replace Gremlin execution through GraphComputer with a new CSR-based implementation.
The important distinction is between VertexProgram and TraversalVertexProgram. VertexProgram remains a useful API for explicitly BSP-oriented algorithms. Users can write programs whose computation occurs at vertices, with messages and global Memory coordinated across supersteps. SparkGraphComputer can continue to provide that distributed programming model while spark-gremlin is deprecated. What I propose removing is the use of GraphComputer as a Gremlin traversal engine. Today, withComputer() converts a traversal into a TraversalVertexProgram. That conversion imposes BSP iterations, message passing, serialization, and distributed Memory coordination on general Gremlin execution. It also introduces different semantics and restrictions from normal Gremlin. Some traversals and steps are unsupported, while others need special handling because side effects and barriers are distributed. Once TraversalVertexProgram is removed, there would no longer be a GraphComputer-based traversal engine. GraphComputer and VertexProgram would remain for pure BSP programming, but Gremlin traversals would no longer be submitted to SparkGraphComputer or another GraphComputer. The replacement for Gremlin OLAP would be "CsrGraph" (which is a placeholder name hence the quotes). "CsrGraph" would be an immutable, reusable graph snapshot based on compressed sparse row storage. It would assign dense ordinals to graph elements and store adjacency, identifiers, labels, properties, and indexes in compact arrays and columns. The snapshot could be stored on disk and memory-mapped so that the operating system can page graph data as needed. "CsrGraph" would support the full Gremlin traversal language over the snapshot rather than the restricted form accepted by TraversalVertexProgram. I prototyped this CSR design to determine whether vertical scaling could cover graphs large enough to make this direction credible. The prototype, across a broad range of Gremlin traversals on a graph with roughly 5M Elements used average memory of about 1/177th of TinkerGraph, 1/186th of TinkerGraphComputer, and 1/9th of Spark, while running 3.7x, 6.5x, and 56x times faster, respectively. In my opinion, these results are strong enough to suggest that this CSR-based path is the right way to move going forward. Removing hadoop-gremlin is a separate but related part of this direction. Despite its name, hadoop-gremlin does not provide a usable Hadoop GraphComputer. MapReduceGraphComputer is an empty placeholder. HadoopGraph is primarily a read-only facade over files, and its OLTP access requires linear scans that our own documentation describes as unreasonable for large graphs. Actual distributed GraphComputer execution is supplied by spark-gremlin. Most importantly, it helps reduce the burden of maintenance which hadoop-gremlin has become especially since its dependencies clash with spark-gremlin. The transition would therefore begin by removing hadoop-gremlin as a public module and artifact in 4.x and deprecating spark-gremlin. SparkGraphComputer would initially retain its existing behavior, including TraversalVertexProgram support. "CsrGraph" could then arrive in a later 4.x release, perhaps 4.1 or 4.2, or in 5.x if it is not ready sooner. TraversalVertexProgram and withComputer()-based Gremlin execution would only be removed after "CsrGraph" provides a viable migration path. At that point, the remaining purpose of spark-gremlin would be distributed execution of VertexProgram for pure BSP workloads. Any questions or concerns? Thanks, Ken
