Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

: Re: [DISCUSS] Upgrade built-in Hive to 2.3.4 I should check the details and feasiablity by myself but to me it sounds fine if it doesn't need extra big efforts. On Tue, 5 Feb 2019, 4:15 am Xiao Li mailto:gatorsm...@gmail.com> wrote: Yes. When our support/integration with Hive 2.x becomes sta

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

>> >>>> >>>> On Fri, Feb 1, 2019 at 2:03 PM Felix Cheung >>>> wrote: >>>> > >>>> > What’s the update and next step on this? >>>> > >>>> > We have real users getting blocked by this issue. >>>

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

ng >>> wrote: >>> > >>> > What’s the update and next step on this? >>> > >>> > We have real users getting blocked by this issue. >>> > >>> > >>> > >>> > From: Xi

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

by this issue. >> > >> > >> > ________________ >> > From: Xiao Li >> > Sent: Wednesday, January 16, 2019 9:37 AM >> > To: Ryan Blue >> > Cc: Marcelo Vanzin; Hyukjin Kwon; Sean Owen; Felix Cheung; Yuming Wang; >>

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

> > > We have real users getting blocked by this issue. > > > > > > > > From: Xiao Li > > Sent: Wednesday, January 16, 2019 9:37 AM > > To: Ryan Blue > > Cc: Marcelo Vanzin; Hyukjin Kwon; Sean Owen; Felix Che

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

t; From: Xiao Li > Sent: Wednesday, January 16, 2019 9:37 AM > To: Ryan Blue > Cc: Marcelo Vanzin; Hyukjin Kwon; Sean Owen; Felix Cheung; Yuming Wang; dev > Subject: Re: [DISCUSS] Upgrade built-in Hive to 2.3.4 > > Thanks for your feedbacks! > > Working with Yuming to reduce the

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

Kwon; Sean Owen; Felix Cheung; Yuming Wang; > dev > *Subject:* Re: [DISCUSS] Upgrade built-in Hive to 2.3.4 > > Thanks for your feedbacks! > > Working with Yuming to reduce the risk of stability and quality. Will keep > you posted when the proposal is ready. > > Cheers

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

] Upgrade built-in Hive to 2.3.4 Thanks for your feedbacks! Working with Yuming to reduce the risk of stability and quality. Will keep you posted when the proposal is ready. Cheers, Xiao Ryan Blue mailto:rb...@netflix.com>> 于2019年1月16日周三 上午9:27写道： +1 for what Marcelo and Hyukji

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

Thanks for your feedbacks! Working with Yuming to reduce the risk of stability and quality. Will keep you posted when the proposal is ready. Cheers, Xiao Ryan Blue 于2019年1月16日周三 上午9:27写道： > +1 for what Marcelo and Hyukjin said. > > In particular, I agree that we can't expect Hive to release

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

+1 for what Marcelo and Hyukjin said. In particular, I agree that we can't expect Hive to release a version that is now more than 3 years old just to solve a problem for Spark. Maybe that would have been a reasonable ask instead of publishing a fork years ago, but I think this is now Spark's

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

+1 to that. HIVE-16391 by itself means we're giving up things like Hadoop 3, and we're also putting the burden on the Hive folks to fix a problem that we created. The current PR is basically a Spark-side fix for that bug. It does mean also upgrading Hive (which gives us Hadoop 3, yay!), but I

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

Resolving HIVE-16391 means Hive to release 1.2.x that contains the fixes of our Hive fork (correct me if I am mistaken). Just to be honest by myself and as a personal opinion, that basically says Hive to take care of Spark's dependency. Hive looks going ahead for 3.1.x and no one would use the

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

It's almost certainly needed just to get off the fork of Hive we're not supposed to have. Yes it's going to impact dependencies, so would need to happen at Spark 3. Separately, its usage could be reduced or removed -- this I don't know much about. But it doesn't really make it harder or easier.

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

If https://issues.apache.org/jira/browse/HIVE-16391 can be resolved, we do not need to keep our fork of Hive. Sean Owen 于2019年1月15日周二 上午10:44写道： > It's almost certainly needed just to get off the fork of Hive we're > not supposed to have. Yes it's going to impact dependencies, so would > need

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

-- >> *From:* Xiao Li >> *Sent:* Tuesday, January 15, 2019 10:03 AM >> *To:* Felix Cheung >> *Cc:* rb...@netflix.com; Yuming Wang; dev >> *Subject:* Re: [DISCUSS] Upgrade built-in Hive to 2.3.4 >> >> Let me take my words back. To read/write a table, Spark users

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

rom the spark core project.. > > > -- > *From:* Xiao Li > *Sent:* Tuesday, January 15, 2019 10:03 AM > *To:* Felix Cheung > *Cc:* rb...@netflix.com; Yuming Wang; dev > *Subject:* Re: [DISCUSS] Upgrade built-in Hive to 2.3.4 > > Let me take my w

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

> >> And we are super 100% dependent on Hive... >> >> >> >> From: Ryan Blue >> Sent: Tuesday, January 15, 2019 9:53 AM >> To: Xiao Li >> Cc: Yuming Wang; dev >> Subject: Re: [DISCUSS] Upgrade bui

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

of it) from the spark core project.. From: Xiao Li Sent: Tuesday, January 15, 2019 10:03 AM To: Felix Cheung Cc: rb...@netflix.com; Yuming Wang; dev Subject: Re: [DISCUSS] Upgrade built-in Hive to 2.3.4 Let me take my words back. To read/write a table, Spark users do

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

ive... > > > -- > *From:* Ryan Blue > *Sent:* Tuesday, January 15, 2019 9:53 AM > *To:* Xiao Li > *Cc:* Yuming Wang; dev > *Subject:* Re: [DISCUSS] Upgrade built-in Hive to 2.3.4 > > How do we know that most Spark users are not using Hive? I w

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

Unless it's going away entirely, and I don't think it is, we at least have to do this to get off the fork of Hive that's being used now. I do think we want to keep Hive from getting into the core though -- see comments on PR. On Tue, Jan 15, 2019 at 11:44 AM Xiao Li wrote: > > Hi, Yuming, > >

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

And we are super 100% dependent on Hive... From: Ryan Blue Sent: Tuesday, January 15, 2019 9:53 AM To: Xiao Li Cc: Yuming Wang; dev Subject: Re: [DISCUSS] Upgrade built-in Hive to 2.3.4 How do we know that most Spark users are not using Hive? I wouldn't

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

. They don’t seem very drastic to me, except for thrift server. Is there another, better approach to thrift server? From: Xiao Li Sent: Tuesday, January 15, 2019 9:44 AM To: Yuming Wang Cc: dev Subject: Re: [DISCUSS] Upgrade built-in Hive to 2.3.4 Hi, Yuming

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

How do we know that most Spark users are not using Hive? I wouldn't be surprised either way, but I do want to make sure we aren't making decisions based on any one person's (or one company's) experience about what "most" Spark users do. On Tue, Jan 15, 2019 at 9:44 AM Xiao Li wrote: > Hi,

Re: [DISCUSS] Upgrade built-in Hive to 2.3.4

Hi, Yuming, Thank you for your contributions! The community aims at reducing the dependence on Hive. Currently, most of Spark users are not using Hive. The changes looks risky to me. To support Hadoop 3.x, we just need to resolve this JIRA: https://issues.apache.org/jira/browse/HIVE-16391

[DISCUSS] Upgrade built-in Hive to 2.3.4

Dear Spark Developers and Users, Hyukjin and I plan to upgrade the built-in Hive from 1.2.1-spark2 to 2.3.4 to solve some critical issues, such as support Hadoop 3.x,