RE: adding shutdownmanagerhook to spark.

2019-05-13 Thread Nasrulla Khan Haris
Thanks Sean Nasrulla -Original Message- From: Sean Owen Sent: Monday, May 13, 2019 4:16 PM To: Nasrulla Khan Haris Cc: dev@spark.apache.org Subject: Re: adding shutdownmanagerhook to spark. Spark just adds a hook to the mechanism that Hadoop exposes. You can do the same. You

Re: adding shutdownmanagerhook to spark.

2019-05-13 Thread Sean Owen
Spark just adds a hook to the mechanism that Hadoop exposes. You can do the same. You shouldn't use Spark's. On Mon, May 13, 2019 at 6:11 PM Nasrulla Khan Haris wrote: > > HI All, > > > > I am trying to add shutdown hook, but looks like shutdown manager object > requires the package to be spark

adding shutdownmanagerhook to spark.

2019-05-13 Thread Nasrulla Khan Haris
HI All, I am trying to add shutdown hook, but looks like shutdown manager object requires the package to be spark only, is there any other API that can help me to do this ? https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala I can see

Re: [VOTE][SPARK-27396] SPIP: Public APIs for extended Columnar Processing Support

2019-05-13 Thread Holden Karau
I’d like to ask this vote period to be extended, I’m interested but I don’t have the cycles to review it in detail and make an informed vote until the 25th. On Tue, May 14, 2019 at 1:49 AM Xiangrui Meng wrote: > My vote is 0. Since the updated SPIP focuses on ETL use cases, I don't > feel

re: vote thread for Avro 1.9.0-RC4

2019-05-13 Thread Michael Heuer
All, FYI, in case you are not also subscribed to d...@avro.apache.org, there is a vote thread currently in progress for Avro version 1.9.0-RC4, which is binary and source incompatible with version 1.8.2 http://people.apache.org/~busbey/avro/1.9.0-RC4/1.8.2_to_1.9.0RC4_compat_report.html

Re: [DISCUSS][SPARK-25299] SPIP: Shuffle storage API

2019-05-13 Thread Yifei Huang (PD)
Hi Mridul - thanks for taking the time to give us feedback! Thoughts on the points that you mentioned: The API is meant to work with the existing SortShuffleManager algorithm. There aren't strict requirements on how other ShuffleManager implementations must behave, so it seems impractical

API for SparkContext ?

2019-05-13 Thread Nasrulla Khan Haris
HI All, Is there a API for sparkContext where we can add our custom code before stopping sparkcontext ? Appreciate your help. Thanks, Nasrulla

Re: [VOTE][SPARK-27396] SPIP: Public APIs for extended Columnar Processing Support

2019-05-13 Thread Xiangrui Meng
My vote is 0. Since the updated SPIP focuses on ETL use cases, I don't feel strongly about it. I would still suggest doing the following: 1. Link the POC mentioned in Q4. So people can verify the POC result. 2. List public APIs we plan to expose in Appendix A. I did a quick check. Beside

Re: [VOTE][SPARK-27396] SPIP: Public APIs for extended Columnar Processing Support

2019-05-13 Thread Thomas graves
It would be nice to get feedback from people who responded on the other vote thread - Reynold, Matei, Xiangrui, does the new version look good? Thanks, Tom On Mon, May 13, 2019 at 8:22 AM Jason Lowe wrote: > > +1 (non-binding) > > Jason > > On Tue, May 7, 2019 at 1:37 PM Thomas graves wrote:

Re: [VOTE][SPARK-27396] SPIP: Public APIs for extended Columnar Processing Support

2019-05-13 Thread Jason Lowe
+1 (non-binding) Jason On Tue, May 7, 2019 at 1:37 PM Thomas graves wrote: > Hi everyone, > > I'd like to call for another vote on SPARK-27396 - SPIP: Public APIs > for extended Columnar Processing Support. The proposal is to extend > the support to allow for more columnar processing. We had