You need to take into consideration 'where' things are executing. The
closure of the 'forEachRDD' executes in the driver. Therefore, the log
statements printed during the execution of that part will be found in the
driver logs.
In contrast, the foreachPartition closure executes on the worker nodes. You
will find the '+++ForEachPartition+++' messages printed in the executor log.
So both statements execute, but in different locations of the distributed
computing environment (aka cluster)
-kr, Gerard.
On Sun, Sep 6, 2015 at 10:53 PM, Alexey Ponkin <alexey.pon...@ya.ru> wrote:
> Hi,
>
> I have the following code
>
> object MyJob extends org.apache.spark.Logging{
> ...
> val source: DStream[SomeType] ...
>
> source.foreachRDD { rdd =>
> logInfo(s"""+++ForEachRDD+++""")
> rdd.foreachPartition { partitionOfRecords =>
> logInfo(s"""+++ForEachPartition+++""")
> }
> }
>
> I was expecting to see both log messages in job log.
> But unfortunately you will never see string '+++ForEachPartition+++' in
> logs, cause block foreachPartition will never execute.
> And also there is no error message or something in logs.
> I wonder is this a bug or known behavior?
> I know that org.apache.spark.Logging is DeveloperAPI, but why it is
> silently fails with no messages?
> What to use instead of org.apache.spark.Logging? in spark-streaming jobs?
>
> P.S. running spark 1.4.1 (on yarn)
>
> Thanks in advance
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
