could not see the print out log in spark functions as mapPartitions

Hi All, I need debug spark job, my general way is to print out the log, however, some bug is in spark functions as mapPartitions etc, and not any log printed from those functionscould be found...Would you help point what is way to the log in the spark own function as mapPartitions? Or, what is

Re: could not see the print out log in spark functions as mapPartitions

Hi Zhiliang, You should be able to see them in the executor logs, which you can view via the Spark UI, in the Executors page (stderr log). HTH, Deng On Tue, Nov 10, 2015 at 11:33 AM, Zhiliang Zhu wrote: > Hi All, > > I need debug spark job, my general way is to

Re: could not see the print out log in spark functions as mapPartitions

Dear Ching-Mallete , There are machines master01, master02 and master03 for the cluster, I could see the stderr log and stdout log in the directories /.../yarn/logs under those machines.In the stderr there are some notes, I just use log to follow the track of the bug, however log in

Re: could not see the print out log in spark functions as mapPartitions

Also for Spark UI , that  is, log from other places could be found, but the log from the functions as mapPartitions could not. On Tuesday, November 10, 2015 11:52 AM, Zhiliang Zhu wrote: Dear Ching-Mallete , There are machines master01, master02 and master03

Re: could not see the print out log in spark functions as mapPartitions

Hi Ching-Mallete, I  have found the log and the reason for that. Thanks a lot!Zhiliang  On Tuesday, November 10, 2015 12:23 PM, Zhiliang Zhu wrote: Also for Spark UI , that  is, log from other places could be found, but the log from the functions as