Github user tejasapatil commented on a diff in the pull request:
https://github.com/apache/spark/pull/18668#discussion_r131187701
--- Diff: docs/configuration.md ---
@@ -2335,5 +2335,61 @@ The location of these configuration files varies
across Hadoop versions, but
a common location is inside of `/etc/hadoop/conf`. Some tools create
configurations on-the-fly, but offer a mechanisms to download copies of
them.
-To make these files visible to Spark, set `HADOOP_CONF_DIR` in
`$SPARK_HOME/spark-env.sh`
+To make these files visible to Spark, set `HADOOP_CONF_DIR` in
`$SPARK_HOME/conf/spark-env.sh`
to a location containing the configuration files.
+
+# Custom Hadoop/Hive Configuration
+
+If your Spark applications interacting with Hadoop, Hive, or both, there
are probably Hadoop/Hive
+configuration files in Spark's class path.
+
+Multiple running applications might require different Hadoop/Hive client
side configurations.
+You can copy and modify `hdfs-site.xml`, `core-site.xml`, `yarn-site.xml`,
`hive-site.xml` in
+Spark's class path for each application, but it is not very convenient and
these
+files are best to be shared with common properties to avoid hard-coding
certain configurations.
+
+The better choice is to use spark hadoop properties in the form of
`spark.hadoop.*`.
+They can be considered as same as normal spark properties which can be set
in `$SPARK_HOME/conf/spark-defalut.conf`
+
+In some cases, you may want to avoid hard-coding certain configurations in
a `SparkConf`. For
+instance, Spark allows you to simply create an empty conf and set
spark/spark hadoop properties.
+
+{% highlight scala %}
+val conf = new SparkConf().set("spark.hadoop.abc.def","xyz")
+val sc = new SparkContext(conf)
+{% endhighlight %}
+
+Also, you can modify or add configurations at runtime:
+{% highlight bash %}
+./bin/spark-submit \
+ --name "My app" \
+ --master local[4] \
+ --conf spark.eventLog.enabled=false \
+ --conf "spark.executor.extraJavaOptions=-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps" \
+ --conf spark.hadoop.abc.def=xyz \
+ myApp.jar
+{% endhighlight %}
+
+## Typical Hadoop/Hive Configurations
--- End diff --
curious : whats the motive behind having this section ? I feel that we
should not get into suggesting these configs external to spark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
