[RESULT][VOTE][SPARK-25299] SPIP: Shuffle Storage API

2019-06-21 Thread Vinoo Ganesh
n open for >72 hours and 3 +1 binding votes have been received, this SPIP passes. Thanks everyone. From: Vinoo Ganesh Date: Friday, June 21, 2019 at 13:44 To: Tom Graves , dhruve ashar , John Zhuge , "Guo, Chenzhao" Cc: Felix Cheung , Yinan Li , "rb...@netflix.com" ,

Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API

2019-06-21 Thread Vinoo Ganesh
+1 binding votes have been received, this SPIP passes. Thanks everyone. From: Tom Graves Date: Friday, June 21, 2019 at 13:02 To: dhruve ashar , John Zhuge , "Guo, Chenzhao" Cc: Vinoo Ganesh , Felix Cheung , Yinan Li , "rb...@netflix.com" , Dongjoon Hyun , Saisai Shao

Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API

2019-06-18 Thread Vinoo Ganesh
ifei Huang (PD)" , Vinoo Ganesh , Imran Rashid Subject: Re: [VOTE][SPARK-25299] SPIP: Shuffle Storage API +1 Glad to see the progress in this space - it’s been more than a year since the original discussion and effort started. From: Yinan Li Sent: Monday, June

Re: Adding Custom finalize method to RDDs.

2019-06-10 Thread Vinoo Ganesh
Generally overriding the finalize() method is an antipattern (it was in fact deprecated in java 11 https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Object.html#finalize()) . What’s the use case here? From: Nasrulla Khan Haris Date: Monday, June 10, 2019 at 15:44 To:

Re: Closing a SparkSession stops the SparkContext

2019-06-05 Thread Vinoo Ganesh
uot; Date: Wednesday, April 3, 2019 at 13:14 To: Vinoo Ganesh Cc: Sean Owen , Arun Mahadevan , "dev@spark.apache.org" Subject: Re: Closing a SparkSession stops the SparkContext For #1, do we agree on the behavior? I think that closing a SparkSession should not close the SparkContext u

Re: Closing a SparkSession stops the SparkContext

2019-04-03 Thread Vinoo Ganesh
needed by one SparkSession and that there isn't any way to clean up now, that's a compelling reason to change the API. Is that the situation? The only downside is making the user separately stop the SparkContext then. From: Vinoo Ganesh Date: Tuesday, April 2, 2019 at 13:24 To: Arun Mahadevan ,

Re: Closing a SparkSession stops the SparkContext

2019-04-02 Thread Vinoo Ganesh
:31 To: Ryan Blue Cc: Vinoo Ganesh , Sean Owen , "dev@spark.apache.org" Subject: Re: Closing a SparkSession stops the SparkContext I am not sure how would it cause a leak though. When a spark session or the underlying context is stopped it should clean up everything. The g

Re: Closing a SparkSession stops the SparkContext

2019-04-02 Thread Vinoo Ganesh
thing else needs to be closed? On Tue, Apr 2, 2019 at 9:45 AM Vinoo Ganesh wrote: > > Hi All - > >I’ve been digging into the code and looking into what appears to be a memory leak (https://urldefense.proofpoint.com/v2/url?u=https-3A__jira.apache.org_jira_brows

Closing a SparkSession stops the SparkContext

2019-04-02 Thread Vinoo Ganesh
Hi All - I’ve been digging into the code and looking into what appears to be a memory leak (https://jira.apache.org/jira/browse/SPARK-27337) and have noticed something kind of peculiar about the way closing a SparkSession is handled. Despite being marked as Closeable, closing/stopping a

Re: time for Apache Spark 3.0?

2018-11-12 Thread Vinoo Ganesh
Makes sense, thanks Reynold. From: Reynold Xin Date: Monday, November 12, 2018 at 16:57 To: Vinoo Ganesh Cc: Xiao Li , Matei Zaharia , Ryan Blue , Mark Hamstra , dev Subject: Re: time for Apache Spark 3.0? Master branch now tracks 3.0.0-SHAPSHOT version, so the next one will be 3.0

Re: time for Apache Spark 3.0?

2018-11-12 Thread Vinoo Ganesh
Quickly following up on this – is there a target date for when Spark 3.0 may be released and/or a list of the likely api breaks that are anticipated? From: Xiao Li Date: Saturday, September 29, 2018 at 02:09 To: Reynold Xin Cc: Matei Zaharia , Ryan Blue , Mark Hamstra ,