Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

[email protected] Wed, 16 Nov 2022 21:22:33 -0800

+1&nbsp;




Ruifeng&nbsp;Zheng
[email protected]



&nbsp;




------------------&nbsp;Original&nbsp;------------------
From:                                                                           
                                             "Wenchen Fan"                      
                                                              
<[email protected]&gt;;
Date:&nbsp;Thu, Nov 17, 2022 10:27 AM
To:&nbsp;"Ye Zhou"<[email protected]&gt;;
Cc:&nbsp;"Jungtaek Lim"<[email protected]&gt;;"Chris 
Nauroth"<[email protected]&gt;;"Maciej"<[email protected]&gt;;"dev"<[email protected]&gt;;
Subject:&nbsp;Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability 
for large applications



+1, I'm looking forward to it!


On Thu, Nov 17, 2022 at 9:44 AM Ye Zhou <[email protected]&gt; wrote:

+1 (non-binding)Thanks for proposing this improvement to SHS, it resolves the 
main performance&nbsp;issue within SHS.


On Wed, Nov 16, 2022 at 1:15 PM Jungtaek Lim <[email protected]&gt; 
wrote:

+1

Nice to see the chance for driver to reduce resource usage and increase 
stability, especially the fact that the driver is SPOF. It's even promising to 
have a future plan to pre-bake the kvstore for SHS from the driver.


Thanks for driving the effort, Gengliang!


On Thu, Nov 17, 2022 at 5:32 AM Chris Nauroth <[email protected]&gt; wrote:

+1 (non-binding)

Gengliang, thank you for the SPIP.

Chris Nauroth








On Wed, Nov 16, 2022 at 4:27 AM Maciej <[email protected]&gt; wrote:

+1
 
 On 11/16/22 13:19, Yuming Wang wrote:
 &gt; +1, non-binding
 &gt; 
 &gt; On Wed, Nov 16, 2022 at 8:12 PM Yang,Jie(INF) <[email protected] 
 &gt; <mailto:[email protected]&gt;&gt; wrote:
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;+1, non-binding____
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;__ __
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;Yang Jie____
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;__ __
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;*??????**: *Mridul Muralidharan <[email protected]
 &gt;&nbsp; &nbsp; &nbsp;<mailto:[email protected]&gt;&gt;
 &gt;&nbsp; &nbsp; &nbsp;*????**: *2022??11??16????????17:35
 &gt;&nbsp; &nbsp; &nbsp;*??????**: *Kent Yao <[email protected] 
<mailto:[email protected]&gt;&gt;
 &gt;&nbsp; &nbsp; &nbsp;*????**: *Gengliang Wang <[email protected]
 &gt;&nbsp; &nbsp; &nbsp;<mailto:[email protected]&gt;&gt;, dev 
<[email protected]
 &gt;&nbsp; &nbsp; &nbsp;<mailto:[email protected]&gt;&gt;
 &gt;&nbsp; &nbsp; &nbsp;*????**: *Re: [VOTE][SPIP] Better Spark UI scalability 
and Driver
 &gt;&nbsp; &nbsp; &nbsp;stability for large applications____
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;__ __
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;__ __
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;+1____
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;__ __
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;Would be great to see history server performance 
improvements and
 &gt;&nbsp; &nbsp; &nbsp;lower resource utilization at driver !____
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;__ __
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;Regards,____
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;Mridul ____
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;__ __
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp;On Wed, Nov 16, 2022 at 2:38 AM Kent Yao 
<[email protected]
 &gt;&nbsp; &nbsp; &nbsp;<mailto:[email protected]&gt;&gt; wrote:____
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;+1, non-binding
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;Gengliang Wang <[email protected] 
<mailto:[email protected]&gt;&gt; ??
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;2022??11??16??????16:36??????
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt;
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; Hi all,
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt;
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; I??d like to start a vote for SPIP: 
"Better Spark UI scalability and Driver stability for large applications"
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt;
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; The goal of the SPIP is to improve 
the Driver's stability by supporting storing Spark's UI data on RocksDB. 
Furthermore, to fasten the read and write operations on RocksDB, it introduces 
a new Protobuf serializer.
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt;
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; Please also refer to the following:
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt;
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; Previous discussion in the dev 
mailing list: [DISCUSS] SPIP: Better Spark UI scalability and Driver stability 
for large applications
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; Design Doc: Better Spark UI 
scalability and Driver stability for large applications
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; JIRA: SPARK-41053
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt;
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt;
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; Please vote on the SPIP for the 
next 72 hours:
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt;
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; [ ] +1: Accept the proposal as an 
official SPIP
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; [ ] +0
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; [ ] -1: I don??t think this is a 
good idea because ??
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt;
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; Kind Regards,
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;&gt; Gengliang
 &gt; 
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; 
&nbsp;---------------------------------------------------------------------
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;To unsubscribe e-mail: 
[email protected]
 &gt;&nbsp; &nbsp; &nbsp; &nbsp; 
&nbsp;<mailto:[email protected]&gt;____
 &gt; 
 
 -- 
 Best regards,
 Maciej Szymkiewicz
 
 Web: https://zero323.net
 PGP: A30CEF0C31A501EC
 
 
 
 



-- 

Zhou, Ye&nbsp; ????

Re: [VOTE][SPIP] Better Spark UI scalability and Driver stability for large applications

Reply via email to