Re: [streaming] Using org.apache.spark.Logging will silently break task execution

You need to take into consideration 'where' things are executing. The closure of the 'forEachRDD' executes in the driver. Therefore, the log statements printed during the execution of that part will be found in the driver logs. In contrast, the foreachPartition closure executes on the worker

Re: [streaming] Using org.apache.spark.Logging will silently break task execution

OK, I got it. When I use 'yarn logs -applicationId ' command everything appears in right place. Thank you! -- Яндекс.Почта — надёжная почта http://mail.yandex.ru/neo2/collect/?exp=1=1 07.09.2015, 01:44, "Gerard Maas" : > You need to take into consideration 'where'

[streaming] Using org.apache.spark.Logging will silently break task execution

Hi, I have the following code object MyJob extends org.apache.spark.Logging{ ... val source: DStream[SomeType] ... source.foreachRDD { rdd => logInfo(s"""+++ForEachRDD+++""") rdd.foreachPartition { partitionOfRecords => logInfo(s"""+++ForEachPartition+++""") } } I