Github user pluradj commented on a diff in the pull request: https://github.com/apache/tinkerpop/pull/721#discussion_r144653511 --- Diff: docs/src/recipes/olap-spark-yarn.asciidoc --- @@ -0,0 +1,153 @@ +//// +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to You under the Apache License, Version 2.0 +(the "License"); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +//// +[[olap-spark-yarn]] +OLAP traversals with Spark on Yarn +---------------------------------- + +TinkerPop's combination of http://tinkerpop.apache.org/docs/current/reference/#sparkgraphcomputer[SparkGraphComputer] +and http://tinkerpop.apache.org/docs/current/reference/#_properties_files[HadoopGraph] allows for running +distributed, analytical graph queries (OLAP) on a computer cluster. The +http://tinkerpop.apache.org/docs/current/reference/#sparkgraphcomputer[reference documentation] covers the cases +where Spark runs locally or where the cluster is managed by a Spark server. However, many users can only run OLAP jobs +via the http://hadoop.apache.org/[Hadoop 2.x] Resource Manager (Yarn), which requires `SparkGraphComputer` to be +configured differently. This recipe describes this configuration. + +Approach +~~~~~~~~ + +Most configuration problems of TinkerPop with Spark on Yarn stem from three reasons: + +1. `SparkGraphComputer` creates its own `SparkContext` so it does not get any configs from the usual `spark-submit` command. +2. The TinkerPop Spark plugin did not include Spark on Yarn runtime dependencies until version 3.2.7/3.3.1. +3. Resolving reason 2 by adding the cluster's `spark-assembly` jar to the classpath creates a host of version +conflicts, because Spark 1.x dependency versions have remained frozen since 2014. + +The current recipe follows a minimalist approach in which no dependencies are added to the dependencies +included in the TinkerPop binary distribution. The Hadoop cluster's Spark installation is completely ignored. This +approach minimizes the chance of dependency version conflicts. + +Prerequisites +~~~~~~~~~~~~~ +This recipe is suitable for both a real external and a local pseudo Hadoop cluster. While the recipe is maintained +for the vanilla Hadoop pseudo-cluster, it has been reported to work on real clusters with Hadoop distributions +from various vendors. + +If you want to try the recipe on a local Hadoop pseudo-cluster, the easiest way to install +it is to look at the install script at https://github.com/apache/tinkerpop/blob/x.y.z/docker/hadoop/install.sh +and the `start hadoop` section of https://github.com/apache/tinkerpop/blob/x.y.z/docker/scripts/build.sh. + +This recipe assumes that you installed the gremlin console with the +http://tinkerpop.apache.org/docs/x.y.z/reference/#spark-plugin[spark plugin] (the +http://tinkerpop.apache.org/docs/x.y.z/reference/#hadoop-plugin[hadoop plugin] is optional). Your Hadoop cluster +may have been configured to use file compression, e.g. lzo compression. If so, you need to copy the relevant +jar (e.g. `hadoop-lzo-*.jar`) to gremlin console's `ext/spark-gremlin/lib` folder. + +For starting the gremlin console in the right environment, create a shell script (e.g. `bin/spark-yarn.sh`) with the --- End diff -- capitalize Gremlin Console throughout the doc
---