Re: DataSourceV2 : Transactional Write support

There might be some help from the staging table catalog as well. -Matt Cheah From: Wenchen Fan Date: Monday, August 5, 2019 at 7:40 PM To: Shiv Prashant Sood Cc: Ryan Blue , Jungtaek Lim , Spark Dev List Subject: Re: DataSourceV2 : Transactional Write support I agree with the temp

Re: DataSourceV2 : Transactional Write support

I agree with the temp table approach. One idea is: maybe we only need one temp table, and each task writes to this temp table. At the end we read the data from the temp table and write it to the target table. AFAIK JDBC can handle concurrent table writing very well, and it's better than creating

Re: Ask for ARM CI for spark

Hi shane, Thanks for your reply. I will wait for you back. ;-) Thanks, Best regards ZhaoBo [image: Mailtrack] Sender notified by Mailtrack

Re: DataSourceV2 : Transactional Write support

Thanks all for the clarification. Regards, Shiv On Sat, Aug 3, 2019 at 12:49 PM Ryan Blue wrote: > > What you could try instead is intermediate output: inserting into > temporal table in executors, and move inserted records to the final table > in driver (must be atomic) > > I think that this

Re: Recognizing non-code contributions

On Mon, Aug 5, 2019 at 9:55 AM Sean Owen wrote: > On Mon, Aug 5, 2019 at 3:50 AM Myrle Krantz wrote: > > So... events coordinators? I'd still make them committers. I guess I'm > still struggling to understand what problem making people VIP's without > giving them committership is trying to

Re: [Discuss] Follow ANSI SQL on table insertion

Ryan, I agree that it needs a VOTE to decide the most reasonable default setting. But what shall we do for Spark 3.0 if there is no more progress on this project anymore(just assume)? In the master branch, we added SQL support for Data Source V2. This exposes a serious problem: the table

Re: [Discuss] Follow ANSI SQL on table insertion

> The choice of whether runtime or analysis time failures should be the default for v2 writes is worth a VOTE on this list. The motivation of my proposal is to make the upgrade from 2.4 to 3.0 more user-friendly and smooth. So I am absolutely +1 on making analysis time failures as default. But

Re: [Discuss] Follow ANSI SQL on table insertion

Wenchen, I don’t think we agree on what “strict mode” would mean. Marco is talking about strict mode as an extension of the flag for throwing exceptions on overflow for decimal operations. That is not ANSI SQL mode. Also, we need more than ANSI SQL and runtime failure modes. For the motivating

Re: Recognizing non-code contributions

On Mon, Aug 5, 2019 at 3:50 AM Myrle Krantz wrote: > So... events coordinators? I'd still make them committers. I guess I'm > still struggling to understand what problem making people VIP's without > giving them committership is trying to solve. We may just agree to disagree, which is fine,

unsubscribe

unsubscribe. - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Recognizing non-code contributions

On Sun, Aug 4, 2019 at 10:06 PM Sean Owen wrote: > Oops, I also failed to copy dev@ > > On Sun, Aug 4, 2019 at 3:06 PM Sean Owen wrote: > > > > On Sun, Aug 4, 2019 at 1:54 PM Myrle Krantz wrote: > > >> No, I think the position here was docs-only contributors _could_ be > > >> committers. The

Re: [Discuss] Follow ANSI SQL on table insertion

I think we need to clarify one thing before further discussion: the proposal is for the next release but not a long term solution. IMO the long term solution should be: completely follow SQL standard (store assignment rule + runtime exception), and provide a variant of functions that can return