Re: Initial Decom PR for Spark 3?

[Hyukjin Kwon]

See https://spark.apache.org/improvement-proposals.html

On Tue, 23 Jun 2020, 07:01 Stephen Boesch,  wrote:

> I guess I missed that "community decision" where the existing design
> document that had been reviewed was put aside and a new SPIP  document was
> required.
>
> On Sun, 21 Jun 2020 at 19:05, Hyukjin Kwon  wrote:
>
>> Yeah, I believe the community decided to do a SPIP for such significant
>> changes. It would be best if we stick to the standard approaches.
>>
>> 2020년 6월 21일 (일) 오전 8:52, Holden Karau 님이 작성:
>>
>>> I believe so, however since Hyukjin is a committer and has asked for an
>>> SPIP I'll be making an SPIP for this next week. I hope to send out the
>>> draft for comment by the end of Spark summit. I'll be using the same design
>>> document for the design component, so if anyone has input on the design
>>> document feel free to start leaving comments there now.
>>>
>>> On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch 
>>> wrote:
>>>
 Hi given there is a design doc (contrary to that common) - is this
 going to move forward?

 On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon  wrote:

> Looks it had to be with SPIP and a proper design doc to discuss.
>
> 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson 님이 작성:
>
>> I'd be willing to pull this in, unless others have concerns post
>> branch-cut.
>>
>> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau 
>> wrote:
>>
>>> Hi Y’all,
>>>
>>> I’ve got a K8s graceful decom PR (
>>> https://github.com/apache/spark/pull/26440
>>>  ) I’d love to try and get in for Spark 3, but I don’t want to push
>>> on it if folks don’t think it’s worth it. I’ve been working on it since
>>> 2017 and it was really close in November but then I had the crash and 
>>> had
>>> to step back for awhile.
>>>
>>> It’s effectiveness is behind a feature flag and it’s been
>>> outstanding for awhile so those points are in its favour. It does 
>>> however
>>> change things in core which is not great.
>>>
>>> Cheers,
>>>
>>> Holden
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>
>>>
>>> --
>>> Twitter: https://twitter.com/holdenkarau
>>> Books (Learning Spark, High Performance Spark, etc.):
>>> https://amzn.to/2MaRAG9  
>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>>
>>

Re: Initial Decom PR for Spark 3?

[Stephen Boesch]

I guess I missed that "community decision" where the existing design
document that had been reviewed was put aside and a new SPIP  document was
required.

On Sun, 21 Jun 2020 at 19:05, Hyukjin Kwon  wrote:

> Yeah, I believe the community decided to do a SPIP for such significant
> changes. It would be best if we stick to the standard approaches.
>
> 2020년 6월 21일 (일) 오전 8:52, Holden Karau 님이 작성:
>
>> I believe so, however since Hyukjin is a committer and has asked for an
>> SPIP I'll be making an SPIP for this next week. I hope to send out the
>> draft for comment by the end of Spark summit. I'll be using the same design
>> document for the design component, so if anyone has input on the design
>> document feel free to start leaving comments there now.
>>
>> On Sat, Jun 20, 2020 at 4:23 PM Stephen Boesch  wrote:
>>
>>> Hi given there is a design doc (contrary to that common) - is this going
>>> to move forward?
>>>
>>> On Thu, 18 Jun 2020 at 18:05, Hyukjin Kwon  wrote:
>>>
 Looks it had to be with SPIP and a proper design doc to discuss.

 2020년 2월 9일 (일) 오전 1:23, Erik Erlandson 님이 작성:

> I'd be willing to pull this in, unless others have concerns post
> branch-cut.
>
> On Tue, Feb 4, 2020 at 2:51 PM Holden Karau 
> wrote:
>
>> Hi Y’all,
>>
>> I’ve got a K8s graceful decom PR (
>> https://github.com/apache/spark/pull/26440
>>  ) I’d love to try and get in for Spark 3, but I don’t want to push
>> on it if folks don’t think it’s worth it. I’ve been working on it since
>> 2017 and it was really close in November but then I had the crash and had
>> to step back for awhile.
>>
>> It’s effectiveness is behind a feature flag and it’s been outstanding
>> for awhile so those points are in its favour. It does however change 
>> things
>> in core which is not great.
>>
>> Cheers,
>>
>> Holden
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9  
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>

CVE-2020-9480: Apache Spark RCE vulnerability in auth-enabled standalone master

[Sean Owen]

Severity: Important

Vendor: The Apache Software Foundation

Versions Affected:
Apache Spark 2.4.5 and earlier

Description:
In Apache Spark 2.4.5 and earlier, a standalone resource manager's master
may
be configured to require authentication (spark.authenticate) via a
shared secret. When enabled, however, a specially-crafted RPC to the
master can succeed in starting an application's resources on the Spark
cluster, even without the shared key. This can be leveraged to execute
shell commands on the host machine.

This does not affect Spark clusters using other resource managers
(YARN, Mesos, etc).


Mitigation:
Users should update to Spark 2.4.6 or 3.0.0.
Where possible, network access to the cluster machines should be
restricted to trusted hosts only.

Credit:
Ayoub Elaassal

References:
https://spark.apache.org/security.html

[Spark Core] Merging PR #23340 for New Executor Memory Metrics

[Alex Scammon]

Hi there devs,

Congrats on Spark 3.0.0, that's great to see.

I'm hoping to get some eyes on something old, however:

  *   https://github.com/apache/spark/pull/23340

I'm really just trying to get some eyes on this PR and see if we can still move 
it forward.  I reached out to the reviewers of the PR but haven't heard 
anything back so I thought I'd try here instead.  We're happy to help sort out 
any remaining issues if there are any.

This particular PR is part of a larger story that LinkedIn was working on here:

  *   https://issues.apache.org/jira/browse/SPARK-23206

Any help getting #23340 opened back up and moving again would be very much 
appreciated.

Cheers,

Alex Scammon
Head of Open Source Engineering
G-Research