On Wed, Feb 4, 2015 at 1:12 PM, Koert Kuipers <ko...@tresata.com> wrote: > about putting stuff on classpath before spark or yarn... yeah you can shoot > yourself in the foot with it, but since the container is isolated it should > be ok, no? we have been using HADOOP_USER_CLASSPATH_FIRST forever with great > success.
The container still has to use Hadoop libraries, e.g., to talk to HDFS. If you override a library it needs with an incompatible one, you may break something. So maybe you've just been lucky. :-) In reality it should be pretty hard to cause breakages if you're careful - e.g., when you override a jar of some library that generally has multiple jars, such as Jackson, you need to include all of them, not just the one(s) you need in your app. MR also has an option similar to Spark's userClassPath (see https://issues.apache.org/jira/browse/MAPREDUCE-1700), which doesn't involve messing with the system's class path. -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
