Re: [DISCUSS] Change default executor log URLs for YARN

For this case I’d agree with Ryan. I haven’t followed this thread and the details of the change since it’s way too much for me to consume “in my free time” (which is 0 nowadays) but I’m pretty sure the existing behavior works for us and very likely we don’t want it to change because of some

Re: [VOTE] Release Apache Spark 2.3.3 (RC2)

Nope, still only 1 binding vote ;) From: Mark Hamstra Sent: Friday, February 8, 2019 7:30 PM To: Marcelo Vanzin Cc: Takeshi Yamamuro; Spark dev list Subject: Re: [VOTE] Release Apache Spark 2.3.3 (RC2) There are 2. C'mon Marcelo, you can make it 3! On Fri, Feb

Re: [VOTE] Release Apache Spark 2.3.3 (RC2)

Thanks for that info, all! Yeah, sure. I'll wait for more votes until the next Wed.(PST) based on the Xiao's suggestion. Enjoy your vacation. Best, Takeshi On Sat, Feb 9, 2019 at 12:30 PM Mark Hamstra wrote: > There are 2. C'mon Marcelo, you can make it 3! > > On Fri, Feb 8, 2019 at 5:03 PM

Re: [VOTE] Release Apache Spark 2.3.3 (RC2)

There are 2. C'mon Marcelo, you can make it 3! On Fri, Feb 8, 2019 at 5:03 PM Marcelo Vanzin wrote: > Hi Takeshi, > > Since we only really have one +1 binding vote, do you want to extend > this vote a bit? > > I've been stuck on a few things but plan to test this (setting things > up now), but

Re: [VOTE] Release Apache Spark 2.3.3 (RC2)

(There isn't a hard time limit on votes; they just need to give at _least_ 72 hours. Indeed just wait until next week for more votes.) On Fri, Feb 8, 2019 at 7:07 PM Xiao Li wrote: > > Hi, Takeshi, > > Many PMCs are on vacation or offsite during this week. If possible, could you > extend it to

Re: [VOTE] Release Apache Spark 2.3.3 (RC2)

Hi, Takeshi, Many PMCs are on vacation or offsite during this week. If possible, could you extend it to next Wed? Happy Lunar New Year! Xiao Marcelo Vanzin 于2019年2月8日周五 下午5:03写道： > Hi Takeshi, > > Since we only really have one +1 binding vote, do you want to extend > this vote a bit? > >

Re: [VOTE] Release Apache Spark 2.3.3 (RC2)

Hi Takeshi, Since we only really have one +1 binding vote, do you want to extend this vote a bit? I've been stuck on a few things but plan to test this (setting things up now), but it probably won't happen before the deadline. On Tue, Feb 5, 2019 at 5:07 PM Takeshi Yamamuro wrote: > > Please

Re: [DISCUSS] Change default executor log URLs for YARN

I'm sure the precondition of changing default behavior is it would help for many users. If not, we may be better to provide alternative approach for minority usage. Maybe we tend to go with option 1, or even force them to use SHS instead (which already enabled the customization of log url). Option

Re: [DISCUSS] Change default executor log URLs for YARN

I'm not sure that many people need this, so it is hard to make a decision. I'm reluctant to change the current behavior if the result is a new papercut to 99% of users and a win for 1%. The suggested change will work for 100% of users, so if we don't want a flag then we should go with that. But I

Re: [VOTE] Release Apache Spark 2.3.3 (RC2)

Sorry for the last minute vote. +1 2019년 2월 8일 (금) 오전 10:15, Takeshi Yamamuro 님이 작성: > Thanks, all. > > Yea, I think we don't need to block the release, too. > > > Jungtaek > Thanks! That is very helpful! > If you find something, please let me know. > > Best, > Takeshi > > On Fri, Feb 8, 2019

Re: [DISCUSS] Change default executor log URLs for YARN

Is a flag needed? You know me, I think flags are often failures of design, or disagreement punted to the user. I can understand retaining old behavior under a flag where the behavior change could be problematic for some users or facilitate migration, but this is just a change to some UI links no?

Re: [DISCUSS] Change default executor log URLs for YARN

I suggest using the current behavior as the default and add a flag to implement the behavior you're suggesting: to link to the logs path in YARN instead of directly to stderr and stdout. On Fri, Feb 8, 2019 at 3:33 PM Jungtaek Lim wrote: > Ryan, > > actually I'm not clear about your suggestion.

Re: [DISCUSS] Change default executor log URLs for YARN

Ryan, actually I'm not clear about your suggestion. For me three possible options here: 1. If we want to let users be able to completely rewrite log urls, that's SPARK-26792 . For SHS we already addressed it. 2. We could let users turning on/off

Re: [DISCUSS] Change default executor log URLs for YARN

Jungtaek, Thanks for the extra context. Those quotes are the confirmation that I was looking for to expose the link you suggest instead of going directly to stderr and stdout. What do you think about my suggestion to change this with a config option? I would prefer that since we use the

Re: [DISCUSS] Change default executor log URLs for YARN

I think that's a reasonable argument, that it provides links to potentially several logs of interest. It reduces the UI clutter a little at the cost of one more hop to get to logs. I don't feel strongly about it but think that's a reasonable thing to do. On Fri, Feb 8, 2019 at 4:57 PM Jungtaek

Re: [DISCUSS] Change default executor log URLs for YARN

Let me quote some voices here: seems like they don't participate this thread. This still doesn't represent the majority are using this pattern, so I'm also OK to make it optional (I might just work on SPARK-26792 to address) and leave the default

building docker images for GPU

Hi spark dev, I created a JIRA issue a while ago ( https://issues.apache.org/jira/browse/SPARK-26398) to add GPU support to Spark docker images, and sent a PR ( https://github.com/apache/spark/pull/23347) that went through several iterations. It was suggested that it should be discussed on the

Re: [DISCUSS] Change default executor log URLs for YARN

Here's what I see from a running job on our cluster. Both of these are links that go to the stderr and stdout links that Spark produces today. stderr : Total file length is 18557 bytes. stdout : Total file length is 0 bytes. While it is nice to see that stderr or stdout has content, I don't

Tungsten Memory Consumer

Hello all,     I am studying about Tungsten Project and I am wondering when Spark creates a Tungsten consumer. While I am running some applications, I see that Spark creates Tungsten Consumer while in other applications not (using the same configuration). When does this happens ? I am