Re: write event logs with YARN

Hi Andrew, Thanks for your explanation, I confirm that the entries show up in the history server UI when I create empty APPLICATION_COMPLETE files for each of them. Christophe. On 03/07/2014 18:27, Andrew Or wrote: Hi Christophe, another Andrew speaking. Your configuration looks fine to me.

Re: write event logs with YARN

Hi Andrew, This does not work (the application failed), I have the following error when I put 3 slashes in the hdfs scheme: (...) Caused by: java.lang.IllegalArgumentException: Pathname

Re: write event logs with YARN

Hi Christophe, another Andrew speaking. Your configuration looks fine to me. From the stack trace it seems that we are in fact closing the file system pre-maturely elsewhere in the system, such that when it tries to write the APPLICATION_COMPLETE file it throws the exception you see. This does

RE: write event logs with YARN

Hi Christophe, Make sure you have 3 slashes in the hdfs scheme. e.g. hdfs:///server_name:9000/user/user_name/spark-events and in the spark-defaults.conf as well.spark.eventLog.dir=hdfs:///server_name:9000/user/user_name/spark-events Date: Thu, 19 Jun 2014 11:18:51 +0200 From: