vinothchandar commented on a change in pull request #937: [HUDI-291] Simplify quickstart documentation URL: https://github.com/apache/incubator-hudi/pull/937#discussion_r331275123
########## File path: docs/quickstart.md ########## @@ -3,196 +3,186 @@ title: Quickstart keywords: hudi, quickstart tags: [quickstart] sidebar: mydoc_sidebar -toc: false +toc: true permalink: quickstart.html --- <br/> -To get a quick peek at Hudi's capabilities, we have put together a [demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) -that showcases this on a docker based setup with all dependent systems running locally. We recommend you replicate the same setup -and run the demo yourself, by following steps [here](docker_demo.html). Also, if you are looking for ways to migrate your existing data to Hudi, -refer to [migration guide](migration_guide.html). -If you have Hive, Hadoop, Spark installed already & prefer to do it on your own setup, read on. +This guide provides a quick peak at Hudi's capabilities using simple spark-shell. Using Spark datasources, this guide +walks through code snippets that allows you to insert and update a Hudi table of default Storage type: + [Copy on Write](https://hudi.apache.org/concepts.html#copy-on-write-storage). +After each write operation we show how to read the data. We will also be looking at how to query a Hudi table incrementally. -## Download Hudi +We have put together a [demo video](https://www.youtube.com/watch?v=VhNgUsxdrD0) that showcases this on a docker based +setup with all dependent systems running locally. We recommend you replicate the same setup and run the demo yourself, +by following steps [here](docker_demo.html). Also, if you are looking for ways to migrate your existing data to Hudi, +refer to [migration guide](migration_guide.html). -Check out [code](https://github.com/apache/incubator-hudi) and normally build the maven project, from command line +For the quickstart, you would need to build Hudi spark bundle jar and provide that to the spark shell as shown below. -``` -$ mvn clean install -DskipTests -DskipITs -``` - -Hudi works with Hive 2.3.x or higher versions. As long as Hive 2.x protocol can talk to Hive 1.x, you can use Hudi to -talk to older hive versions. - -For IDE, you can pull in the code into IntelliJ as a normal maven project. -You might want to add your spark jars folder to project dependencies under 'Module Setttings', to be able to run from IDE. - - -### Version Compatibility +## Build Hudi spark bundle jar -Hudi requires Java 8 to be installed on a *nix system. Hudi works with Spark-2.x versions. -Further, we have verified that Hudi works with the following combination of Hadoop/Hive/Spark. - -| Hadoop | Hive | Spark | Instructions to Build Hudi | -| ---- | ----- | ---- | ---- | -| Apache hadoop-2.[7-8].x | Apache hive-2.3.[1-3] | spark-2.[1-3].x | Use "mvn clean install -DskipTests" | - -If your environment has other versions of hadoop/hive/spark, please try out Hudi -and let us know if there are any issues. - -## Generate Sample Dataset - -### Environment Variables - -Please set the following environment variables according to your setup. We have given an example setup with CDH version +Hudi requires Java 8 to be installed on a *nix system. +Check out [code](https://github.com/apache/incubator-hudi) and normally build the maven project, from command line: ``` -cd incubator-hudi -export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/ -export HIVE_HOME=/var/hadoop/setup/apache-hive-1.1.0-cdh5.7.2-bin -export HADOOP_HOME=/var/hadoop/setup/hadoop-2.6.0-cdh5.7.2 -export HADOOP_INSTALL=/var/hadoop/setup/hadoop-2.6.0-cdh5.7.2 -export HADOOP_CONF_DIR=$HADOOP_INSTALL/etc/hadoop -export SPARK_HOME=/var/hadoop/setup/spark-2.3.1-bin-hadoop2.7 -export SPARK_INSTALL=$SPARK_HOME -export SPARK_CONF_DIR=$SPARK_HOME/conf -export PATH=$JAVA_HOME/bin:$HIVE_HOME/bin:$HADOOP_HOME/bin:$SPARK_INSTALL/bin:$PATH +$ mvn clean install -DskipTests -DskipITs + +$ # Export the location of hudi-spark-bundle for later reference +$ mkdir -p /var/tmp/hudi && cp packaging/hudi-spark-bundle/target/hudi-spark-bundle-*.*.*-SNAPSHOT.jar /var/tmp/hudi/hudi-spark-bundle.jar +$ export HUDI_SPARK_BUNDLE_PATH=/var/tmp/hudi/hudi-spark-bundle.jar ``` -### Run HoodieJavaApp +## Setup spark-shell +Hudi works with Spark-2.x versions. You can follow instructions [here](https://spark.apache.org/downloads.html) for +setting up spark. -Run __hudi-spark/src/test/java/HoodieJavaApp.java__ class, to place a two commits (commit 1 => 100 inserts, commit 2 => 100 updates to previously inserted 100 records) onto your DFS/local filesystem. Use the wrapper script -to run from command-line +From the extracted directory run spark-shell with Hudi as: ``` -cd hudi-spark -./run_hoodie_app.sh --help -Usage: <main class> [options] - Options: - --help, -h - Default: false - --table-name, -n - table name for Hudi sample table - Default: hoodie_rt - --table-path, -p - path for Hudi sample table - Default: file:///tmp/hoodie/sample-table - --table-type, -t - One of COPY_ON_WRITE or MERGE_ON_READ - Default: COPY_ON_WRITE +bin/spark-shell --jars $HUDI_SPARK_BUNDLE_PATH --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' ``` -The class lets you choose table names, output paths and one of the storage types. In your own applications, be sure to include the `hudi-spark` module as dependency -and follow a similar pattern to write/read datasets via the datasource. - -## Query a Hudi dataset - -Next, we will register the sample dataset into Hive metastore and try to query using [Hive](#hive), [Spark](#spark) & [Presto](#presto) +Setup table name, base path and a data generator to generate records for this guide. -### Start Hive Server locally +``` +import org.apache.hudi.QuickstartUtils._ +import scala.collection.JavaConversions._ +import org.apache.spark.sql.SaveMode._ +import org.apache.hudi.DataSourceWriteOptions._ +import org.apache.hudi.config.HoodieWriteConfig._ +import org.apache.hudi.DataSourceReadOptions._ +val tableName = "hudi_cow_table1" +val basePath = "/tmp/hudi_cow_table1" Review comment: nit.. explicitly specify `file:///tmp` since if they had hadoop in class path somehow, it could end up there ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services