Ottomata has uploaded a new change for review. https://gerrit.wikimedia.org/r/192870
Change subject: Fix to allow spark to load native Hadoop libs ...................................................................... Fix to allow spark to load native Hadoop libs This fixes WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable on startup. See: http://apache-spark-user-list.1001560.n3.nabble.com/LZO-support-in-Spark-1-0-0-nothing-seems-to-work-td14494.html Change-Id: I847840681dd27df5a8b6af9f05e726d9a5c007f5 --- M templates/spark/spark-env.sh.erb 1 file changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/cdh refs/changes/70/192870/1 diff --git a/templates/spark/spark-env.sh.erb b/templates/spark/spark-env.sh.erb index e851e4d..dc2ecc8 100755 --- a/templates/spark/spark-env.sh.erb +++ b/templates/spark/spark-env.sh.erb @@ -86,3 +86,7 @@ ### you want to run with scala version, that is included with the package #export SCALA_HOME=${SCALA_HOME:-/usr/lib/spark/scala} #export PATH=$PATH:$SCALA_HOME/bin + +# pyspark and spark-shell don't pick up Hadoop native libs on their own. +# See: http://apache-spark-user-list.1001560.n3.nabble.com/LZO-support-in-Spark-1-0-0-nothing-seems-to-work-td14494.html +export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/hadoop/lib/native -- To view, visit https://gerrit.wikimedia.org/r/192870 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I847840681dd27df5a8b6af9f05e726d9a5c007f5 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet/cdh Gerrit-Branch: master Gerrit-Owner: Ottomata <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
