Hello, folks.
We just recently switched to using Yarn on our cluster (when upgrading to
cloudera 5.4.1)
I'm trying to run a spark job from within a broader application (a web
service running on Jetty), so I can't just start it using spark-submit.
Does anyone know of an instructions page on how to do that under yarn?
I've managed to get it mostly there by including all spark, yarn, hadoop,
and hdfs config files in my SparkConf (somewhat indirectly, and that is a
bit of a short-hand), but while the job shows up now under yarn, and has
its own applications web ui page, it's not showing up under the main spark
page, and it's still missing some things (like it can't find the native
library for snappy compression), so I still think I'm doing something wrong.
Any help or hints would be much appreciated.
Thanks,
-Nathan
