Re: [DISCUSS] Change default executor log URLs for YARN

behavior doesn’t seem broken... > > > > I could be wrong though. > > > > > > > > From: Ryan Blue > > Sent: Friday, February 8, 2019 4:39 PM > > To: Sean Owen > > Cc: Jungtaek Lim; dev > > Subject: Re: [DISCUSS] Change default executor log

Re: [DISCUSS] Change default executor log URLs for YARN

> > I could be wrong though. > > > > From: Ryan Blue > Sent: Friday, February 8, 2019 4:39 PM > To: Sean Owen > Cc: Jungtaek Lim; dev > Subject: Re: [DISCUSS] Change default executor log URLs for YARN > > I'm not sure that many people need thi

Re: [DISCUSS] Change default executor log URLs for YARN

: Jungtaek Lim; dev Subject: Re: [DISCUSS] Change default executor log URLs for YARN I'm not sure that many people need this, so it is hard to make a decision. I'm reluctant to change the current behavior if the result is a new papercut to 99% of users and a win for 1%. The suggested change

Re: [DISCUSS] Change default executor log URLs for YARN

I'm sure the precondition of changing default behavior is it would help for many users. If not, we may be better to provide alternative approach for minority usage. Maybe we tend to go with option 1, or even force them to use SHS instead (which already enabled the customization of log url). Option

Re: [DISCUSS] Change default executor log URLs for YARN

I'm not sure that many people need this, so it is hard to make a decision. I'm reluctant to change the current behavior if the result is a new papercut to 99% of users and a win for 1%. The suggested change will work for 100% of users, so if we don't want a flag then we should go with that. But I

Re: [DISCUSS] Change default executor log URLs for YARN

Is a flag needed? You know me, I think flags are often failures of design, or disagreement punted to the user. I can understand retaining old behavior under a flag where the behavior change could be problematic for some users or facilitate migration, but this is just a change to some UI links no?

Re: [DISCUSS] Change default executor log URLs for YARN

I suggest using the current behavior as the default and add a flag to implement the behavior you're suggesting: to link to the logs path in YARN instead of directly to stderr and stdout. On Fri, Feb 8, 2019 at 3:33 PM Jungtaek Lim wrote: > Ryan, > > actually I'm not clear about your suggestion.

Re: [DISCUSS] Change default executor log URLs for YARN

Ryan, actually I'm not clear about your suggestion. For me three possible options here: 1. If we want to let users be able to completely rewrite log urls, that's SPARK-26792 . For SHS we already addressed it. 2. We could let users turning on/off

Re: [DISCUSS] Change default executor log URLs for YARN

Jungtaek, Thanks for the extra context. Those quotes are the confirmation that I was looking for to expose the link you suggest instead of going directly to stderr and stdout. What do you think about my suggestion to change this with a config option? I would prefer that since we use the

Re: [DISCUSS] Change default executor log URLs for YARN

I think that's a reasonable argument, that it provides links to potentially several logs of interest. It reduces the UI clutter a little at the cost of one more hop to get to logs. I don't feel strongly about it but think that's a reasonable thing to do. On Fri, Feb 8, 2019 at 4:57 PM Jungtaek

Re: [DISCUSS] Change default executor log URLs for YARN

Let me quote some voices here: seems like they don't participate this thread. This still doesn't represent the majority are using this pattern, so I'm also OK to make it optional (I might just work on SPARK-26792 to address) and leave the default

Re: [DISCUSS] Change default executor log URLs for YARN

Here's what I see from a running job on our cluster. Both of these are links that go to the stderr and stdout links that Spark produces today. stderr : Total file length is 18557 bytes. stdout : Total file length is 0 bytes. While it is nice to see that stderr or stdout has content, I don't

Re: [DISCUSS] Change default executor log URLs for YARN

New URL shows all of local logs which includes stdout and stderr as a list. The change would help when end users modify their log4j configuration to have another log files, as well as GC logs. Currently Spark only shows two static files (stdout, stderr) as individual links so easier to see the

Re: [DISCUSS] Change default executor log URLs for YARN

Jungtaek, What is shown at the new URL and how would this improve usability? On Thu, Feb 7, 2019 at 12:45 AM Jungtaek Lim wrote: > Hi devs, > > Based on the suggestion Tom Graves gave me in SPARK-26792 > , I'd like to hear > voices on changing

[DISCUSS] Change default executor log URLs for YARN

Hi devs, Based on the suggestion Tom Graves gave me in SPARK-26792 , I'd like to hear voices on changing default executor log URLs for YARN, specifically removing "stdout" and "stderr" and provide link which shows log file"s". For example,