Re: [Announce] New committer: Peter Vary

[Jimmy Xiang]

Congrats!!

On Fri, Jul 7, 2017 at 11:38 AM, Sergio Pena  wrote:
> Congrats Peter !!
>
> On Fri, Jul 7, 2017 at 12:51 PM, Vaibhav Gumashta > wrote:
>
>> Congratulations Peter!
>>
>> On 7/7/17, 10:46 AM, "Vineet Garg"  wrote:
>>
>> >Congrats Peter!
>> >
>> >> On Jul 7, 2017, at 9:27 AM, Xuefu Zhang  wrote:
>> >>
>> >> Congratulations!
>> >>
>> >> On Fri, Jul 7, 2017 at 4:17 AM, Adam Szita  wrote:
>> >>
>> >>> Congrats all!
>> >>>
>> >>> On 7 July 2017 at 10:03, Zoltan Haindrich 
>> >>> wrote:
>> >>>
>>  Congratulations Peter, Teddy, Deepesh, Vihang and Sahil!
>>  It's great to see that the Hive community is growing!
>> 
>>  On 6 Jul 2017 02:52, Ashutosh Chauhan  wrote:
>>  The Project Management Committee (PMC) for Apache Hive has invited
>> Peter
>>  Vary to become a committer and we are pleased to announce that he has
>>  accepted.
>> 
>>  Welcome, Peter!
>> 
>>  Thanks,
>>  Ashutosh
>> 
>> 
>> >>>
>> >
>>
>>

Re: [ANNOUNCE] New PMC Member : Matt McCline

[Jimmy Xiang]

Congrats!!

On Fri, Jul 7, 2017 at 11:37 AM, Sergio Pena  wrote:
> Congrats Matt !!
>
> On Fri, Jul 7, 2017 at 12:51 PM, Vaibhav Gumashta > wrote:
>
>> Congratulations Matt!
>>
>> On 7/7/17, 10:46 AM, "Vineet Garg"  wrote:
>>
>> >Congrats Matt!
>> >> On Jul 7, 2017, at 9:57 AM, Vihang Karajgaonkar 
>> >>wrote:
>> >>
>> >> Congratulations Matt!
>> >>
>> >> On Fri, Jul 7, 2017 at 9:27 AM, Xuefu Zhang  wrote:
>> >>
>> >>> Congratulations!
>> >>>
>> >>> On Fri, Jul 7, 2017 at 8:27 AM, Eugene Koifman
>> >>>
>> >>> wrote:
>> >>>
>>  Congratulations!
>> 
>>  On 7/7/17, 1:04 AM, "Zoltan Haindrich" 
>> >>> wrote:
>> 
>> Congrats Matt!
>> 
>> On 7 Jul 2017 09:46, Peter Vary  wrote:
>> Congratulations Matt! :)
>> 
>> 2017. júl. 7. 0:34 ezt írta ("Jesus Camacho Rodriguez" <
>>  jcama...@apache.org
>> > ):
>> 
>> > Congrats Matt!
>> >
>> > -Jesús
>> >
>> >
>> >
>> > On 7/6/17, 11:13 PM, "Lefty Leverenz" 
>>  wrote:
>> >
>> >> Congratulations Matt!  Well deserved.
>> >>
>> >> -- Lefty
>> >>
>> >> On Thu, Jul 6, 2017 at 11:31 AM, Ashutosh Chauhan <
>>  hashut...@apache.org>
>> >> wrote:
>> >>
>> >>> On behalf of the Hive PMC I am delighted to announce Matt
>> >>> McCline
>>  is
>> >>> joining Hive PMC.
>> >>> Matt is a long time contributor in Hive and is focusing on
>>  vectorization
>> >>> these days.
>> >>>
>> >>> Welcome, Matt!
>> >>>
>> >>> Thanks,
>> >>> Ashutosh
>> >>>
>> >
>> >
>> 
>> 
>> 
>> 
>> >>>
>> >
>>
>>

Re: [Announce] New committer: Teddy Choi

[Jimmy Xiang]

Congrats!!

On Fri, Jul 7, 2017 at 11:37 AM, Sergio Pena  wrote:
> Congratulations Teddy!
>
> On Fri, Jul 7, 2017 at 12:51 PM, Vaibhav Gumashta > wrote:
>
>> Congratulations Teddy!
>>
>> On 7/7/17, 10:46 AM, "Vineet Garg"  wrote:
>>
>> >Congratulations!
>> >
>> >> On Jul 7, 2017, at 10:25 AM, Gunther Hagleitner
>> >> wrote:
>> >>
>> >> Congrats Teddy!
>> >> 
>> >> From: Xuefu Zhang 
>> >> Sent: Friday, July 07, 2017 9:26 AM
>> >> To: dev@hive.apache.org
>> >> Subject: Re: [Announce] New committer: Teddy Choi
>> >>
>> >> Congratulations!
>> >>
>> >> On Fri, Jul 7, 2017 at 12:09 AM, Matthew McCline
>> >>
>> >> wrote:
>> >>
>> >>> Congratulations Teddy!
>> >>>
>> >>> Get Outlook for iOS
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Jul 5, 2017 at 5:53 PM -0700, "Ashutosh Chauhan" <
>> >>> hashut...@apache.org> wrote:
>> >>>
>> >>>
>> >>> The Project Management Committee (PMC) for Apache Hive has invited
>> >>>Teddy
>> >>> Choi to become a committer and we are pleased to announce that he has
>> >>> accepted.
>> >>>
>> >>> Welcome, Teddy!
>> >>>
>> >>> Thanks,
>> >>> Ashutosh
>> >>>
>> >>>
>> >>
>> >>
>> >
>>
>>

Re: [DISCUSS] Separating out the metastore as its own TLP

[Jimmy Xiang]

Yeah, this is good idea. +1

On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun  wrote:
> HMS has become the shared catalog service for multiple projects outside
> Hive,
> so +1 on this move (and maybe a different project name?).
>
> On Fri, Jun 30, 2017 at 2:10 PM, Owen O'Malley 
> wrote:
>
>> I'm +1 on separating out the metastore. It recognizes the reality that a
>> lot of different projects use the Hive Metastore and opening up the
>> community is a great move.
>>
>> ..Owen
>>
>> On Fri, Jun 30, 2017 at 1:30 PM, Xuefu Zhang  wrote:
>>
>> > +1, sounds like a good idea!
>> >
>> > On Fri, Jun 30, 2017 at 1:24 PM, Harsha  wrote:
>> >
>> > > Thanks for the proposal Alan. I am +1 on separating the Hive Metastore.
>> > > This is a great opportunity for building a Metastore to not only
>> address
>> > > schemas for the data at rest but also for the data in motion. We have a
>> > > SchemaRegistry (http://github.com/hortonworks/registry)  project that
>> > > allows users to register schemas for data in motion and integrates with
>> > > Kafka, Kinesis, Evenhubs and other messaging queues. This will provide
>> > > us with opportunity to integrate our apis with Hive Metastore and
>> > > provide with one project that is truly a single metastore that can hold
>> > > all schemas.
>> > >
>> > > Thanks,
>> > > Harsha
>> > >
>> > > On Fri, Jun 30, 2017, at 01:18 PM, Sergio Pena wrote:
>> > > > Great, thanks Alan for putting all this in the email.
>> > > > +1
>> > > >
>> > > > Allowing other components to continue to use the Metastore without
>> the
>> > > > need
>> > > > to use Hive dependencies is a big plus for them. I agree with
>> > everything
>> > > > you mention on the email.
>> > > >
>> > > > - Sergio
>> > > >
>> > > > On Fri, Jun 30, 2017 at 1:49 PM, Julian Hyde 
>> wrote:
>> > > >
>> > > > > +1
>> > > > >
>> > > > > As a Calcite PMC member, I am very pleased to see this change.
>> > Calcite
>> > > > > reads metadata from a variety of sources (including JDBC databases,
>> > > NoSQL
>> > > > > databases such as Cassandra and Druid, and streaming systems), and
>> if
>> > > more
>> > > > > of those sources choose to store their metadata in the metastore it
>> > > will
>> > > > > make our lives easier.
>> > > > >
>> > > > > Hive’s metastore has established a position as the place to go for
>> > > > > metadata in the Hadoop ecosystem. Not all metadata is relational,
>> or
>> > > > > processed by Hive, so there are other parties using the metastore
>> who
>> > > > > justifiably would like to influence its direction. Opening up the
>> > > metastore
>> > > > > will help retain and extend this position.
>> > > > >
>> > > > > Julian
>> > > > >
>> > > > >
>> > > > > On 2017-06-30 10:00 (-0700), "Dimitris ts...@apache.org> wrote:
>> > > > > >
>> > > > > >
>> > > > > > On 2017-06-30 07:56 (-0700), Alan Gates 
>> wrote: >
>> > > > > > > A few of us have been talking and come to the conclussion that
>> it
>> > > > > would be>
>> > > > > > > a good thing to split out the Hive metastore into its own
>> Apache
>> > > > > project.>
>> > > > > > > Below and in the linked wiki page we explain what we see as the
>> > > > > advantages>
>> > > > > > > to this and how we would go about it.>
>> > > > > > > >
>> > > > > > > Hive’s metastore has long been used by other projects in the
>> > > Hadoop>
>> > > > > > > ecosystem to store and access metadata.  Apache Impala, Apache
>> > > Spark,>
>> > > > > > > Apache Drill, Presto, and other systems all use Hive’s
>> metastore.
>> > > > > Some,>
>> > > > > > > like Impala and Presto can use it as their own metadata system
>> > with
>> > > > > the>
>> > > > > > > rest of Hive not present.>
>> > > > > > > >
>> > > > > > > This sharing is excellent for the ecosystem.  Together with
>> HDFS
>> > it
>> > > > > allows>
>> > > > > > > users to use the tool of their choice while still accessing the
>> > > same
>> > > > > shared>
>> > > > > > > data.  But having this shared metadata inside the Hive project
>> > > limits
>> > > > > the>
>> > > > > > > ability of other projects to contribute to the metastore.  It
>> > also
>> > > > > makes it>
>> > > > > > > harder for new systems that have similar but not identical
>> > > metadata>
>> > > > > > > requirements (for example, stream processing systems on top of
>> > > Apache>
>> > > > > > > Kafka) to use Hive’s metastore.  This difficulty for other
>> > systems
>> > > > > comes>
>> > > > > > > out in two ways.  One, it is hard for non-Hive community
>> members
>> > > to>
>> > > > > > > participate in the project.  Second, it adds operational cost
>> > since
>> > > > > users>
>> > > > > > > are forced to deploy all of the Hive jars just to get the
>> > > metastore to
>> > > > > work.>
>> > > > > > > >
>> > > > > > > Therefore we propose to split Hive’s metastore out into a
>> > separate
>> > > > > Apache>
>> > > > > > > project.  This new project will continue to support the same
>> > Thrift
>> > > > > API as>
>> > > > > > > the current metastore.  It will continue to

Re: [Announce] New PMC members

[Jimmy Xiang]

Congrats!!

On Fri, May 26, 2017 at 2:52 AM, Adam Szita  wrote:
> Congrats to all, well done!
>
> Adam
>
> On 26 May 2017 at 08:17, Anshuman Dwivedi  wrote:
>
>> Congrats !
>>
>> Regards,
>> Anshuman Dwivedi
>>
>>
>> -Peter Vary  wrote: -
>> To: dev@hive.apache.org
>> From: Peter Vary 
>> Date: 05/26/2017 10:57AM
>> Subject: Re: [Announce] New PMC members
>>
>> Wow!
>> That's a spring shower of PMCs. :)
>> Well deserved Yongzhi, Daniel, Vaibhav, Sergio, Aihua, Chaoyu!
>>
>> Congratulations to all of you!
>>
>> Peter
>>
>> 2017. máj. 26. 6:42 ezt írta ("Ashutosh Chauhan" ):
>>
>> The Project Management Committee (PMC) for Apache Hive has invited Yongzhi
>> Chen to become a PMC member and we are pleased to announce that he has
>> accepted.
>>
>> Please join me in congratulating Yongzhi!
>>
>> Thanks,
>> Ashutosh on behalf of Hive PMC
>> =-=-=
>> Notice: The information contained in this e-mail
>> message and/or attachments to it may contain
>> confidential or privileged information. If you are
>> not the intended recipient, any dissemination, use,
>> review, distribution, printing or copying of the
>> information contained in this e-mail message
>> and/or attachments to it are strictly prohibited. If
>> you have received this communication in error,
>> please notify us by reply e-mail or telephone and
>> immediately and permanently delete the message
>> and any attachments. Thank you
>>
>>
>>

Re: Welcome Rui Li to Hive PMC

[Jimmy Xiang]

Congrats! Rui!!

On Thu, May 25, 2017 at 8:29 AM, Anshuman Dwivedi
 wrote:
> Congrats !
>
> Regards,
> Anshuman Dwivedi
>
> -Sergio Pena  wrote: -
> To: dev 
> From: Sergio Pena 
> Date: 05/25/2017 08:57PM
> Subject: Re: Welcome Rui Li to Hive PMC
>
> Congratulations Rui !!!
>
> On Thu, May 25, 2017 at 4:44 AM, Rui Li  wrote:
>
>> Thank you guys :)
>>
>> On Thu, May 25, 2017 at 3:29 PM, Peter Vary  wrote:
>>
>> > Congratulations Rui!
>> >
>> > > On May 25, 2017, at 6:19 AM, Xuefu Zhang  wrote:
>> > >
>> > > Hi all,
>> > >
>> > > It's an honer to announce that Apache Hive PMC has recently voted to
>> > invite
>> > > Rui Li as a new Hive PMC member. Rui is a long time Hive contributor
>> and
>> > > committer, and has made significant contribution in Hive especially in
>> > Hive
>> > > on Spark. Please join me in congratulating him and looking forward to a
>> > > bigger role that he will play in Apache Hive project.
>> > >
>> > > Thanks,
>> > > Xuefu
>> >
>> >
>>
>>
>> --
>> Best regards!
>> Rui Li
>> Cell: (+86) 13564950210
>>
> =-=-=
> Notice: The information contained in this e-mail
> message and/or attachments to it may contain
> confidential or privileged information. If you are
> not the intended recipient, any dissemination, use,
> review, distribution, printing or copying of the
> information contained in this e-mail message
> and/or attachments to it are strictly prohibited. If
> you have received this communication in error,
> please notify us by reply e-mail or telephone and
> immediately and permanently delete the message
> and any attachments. Thank you
>
>

Re: [Announce] New committer: Vineet Garg

[Jimmy Xiang]

Congrats!!

On Tue, May 9, 2017 at 1:47 PM, Sergio Pena  wrote:
> Congratulations Vinnet !!
>
> On Tue, May 9, 2017 at 3:42 PM, Wei Zheng  wrote:
>
>> Congrats!
>>
>> Thanks,
>> Wei
>>
>> On 5/9/17, 13:36, "Gunther Hagleitner" 
>> wrote:
>>
>> Congrats Vineet!
>> 
>> From: Ashutosh Chauhan 
>> Sent: Tuesday, May 09, 2017 1:24 PM
>> To: dev@hive.apache.org
>> Subject: [Announce] New committer: Vineet Garg
>>
>> The Project Management Committee (PMC) for Apache Hive has invited
>> Vineet
>> Garg to become a committer and we are pleased to announce that he has
>> accepted.
>>
>> Welcome, Vineet!
>>
>> Thanks,
>> Ashutosh
>>
>>
>>
>>

Re: Welcome new Hive committer, Zhihai Xu

[Jimmy Xiang]

Congrats!!

On Fri, May 5, 2017 at 10:15 AM, Chinna Rao Lalam
 wrote:
> Congratulations Zhihai...
>
> On Fri, May 5, 2017 at 10:22 PM, Xuefu Zhang  wrote:
>>
>> Hi all,
>>
>> I'm very please to announce that Hive PMC has recently voted to offer
>> Zhihai a committership which he accepted. Please join me in congratulating
>> on this recognition and thanking him for his contributions to Hive.
>>
>> Regards,
>> Xuefu
>
>
>
>
> --
> Hope It Helps,
> Chinna

[jira] [Created] (HIVE-16348) HoS query is canceled but error message shows RPC is closed

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-16348:
--

 Summary: HoS query is canceled but error message shows RPC is 
closed
 Key: HIVE-16348
 URL: https://issues.apache.org/jira/browse/HIVE-16348
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


When a HoS query is interrupted in getting app id, it keeps trying to get 
status till timedout, and return some RPC is closed error message which is 
misleading.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16292) SparkUtilities upload to HDFS doesn't work with viewfs

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-16292:
--

 Summary: SparkUtilities upload to HDFS doesn't work with viewfs
 Key: HIVE-16292
 URL: https://issues.apache.org/jira/browse/HIVE-16292
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


If scratchdir is set to viewfs, HoS fails with exception like

{noformt}
java.lang.IllegalArgumentException: Wrong FS: 
viewfs://ns-default/tmp/hive_scratch/hive/hive/_spark_session_dir/f4031fca-2885-4e7a-9b05-764d25d0e488/hive-exec-1.1.0-cdh5.7.2.jar,
 expected: hdfs://nameservice1
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:657)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:194)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:106)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:499)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:351)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
at 
org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1949)
at 
org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.uploadToHDFS(SparkUtilities.java:86)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16286) Log canceled query id

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-16286:
--

 Summary: Log canceled query id
 Key: HIVE-16286
 URL: https://issues.apache.org/jira/browse/HIVE-16286
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Trivial


Currently, just a generic message is logged when a query is canceled. It is 
better to log the query id as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [ANNOUNCE] New PMC Member : Eugene Koifman

[Jimmy Xiang]

Congrats!!

On Wed, Mar 15, 2017 at 7:31 AM, Sergio Pena  wrote:
> Congratulations Eugene !!
>
> On Wed, Mar 15, 2017 at 5:41 AM, Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
>
>> Congratulations Eugene!
>>
>> Thanks
>> Prasanth
>>
>>
>>
>>
>> On Tue, Mar 14, 2017 at 10:02 PM -1000, "Zoltan Haindrich" <
>> zhaindr...@hortonworks.com> wrote:
>>
>>
>> Congrats Eugene!!
>>
>> On 15 Mar 2017 07:50, Peter Vary  wrote:
>> Congratulations! :)
>>
>> 2017. márc. 15. 7:05 ezt írta ("Vaibhav Gumashta" ):
>>
>> > Congrats Eugene!
>> >
>> >
>> > On 3/14/17, 11:03 PM, "Rajesh Balamohan"  wrote:
>> >
>> > >Congrats Eugene!! :)
>> > >
>> > >~Rajesh.B
>> > >
>> > >On Wed, Mar 15, 2017 at 11:21 AM, Pengcheng Xiong
>> > >wrote:
>> > >
>> > >> Congrats! Well deserved!
>> > >>
>> > >> Thanks.
>> > >> Pengcheng
>> > >>
>> > >> On Tue, Mar 14, 2017 at 10:39 PM, Ashutosh Chauhan
>> > >>
>> > >> wrote:
>> > >>
>> > >> > On behalf of the Hive PMC I am delighted to announce Eugene Koifman
>> is
>> > >> > joining Hive PMC.
>> > >> > Eugene is a long time contributor in Hive and is focusing on ACID
>> > >>support
>> > >> > areas these days.
>> > >> >
>> > >> > Welcome, Eugene!
>> > >> >
>> > >> > Thanks,
>> > >> > Ashutosh
>> > >> >
>> > >>
>> >
>> >
>>
>>
>>

Re: [ANNOUNCE] New committer: Zoltan Haindrich

[Jimmy Xiang]

Congrats, Zoltan!!

On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan  wrote:
> Congrats, Zoltan!
>
> Welcome aboard. :)
>
> On Feb 21, 2017 15:42, "Rajesh Balamohan"  wrote:
>
>> Congrats Zoltan. :)
>>
>> ~Rajesh.B
>>
>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng  wrote:
>>
>> > Congrats Zoltan!
>> >
>> > Thanks,
>> > Wei
>> >
>> > On 2/21/17, 13:09, "Alan Gates"  wrote:
>> >
>> > On behalf of the Hive PMC I am happy to announce Zoltan Haindrich is
>> > our newest committer.  He has been contributing to Hive for several
>> months
>> > across a number of areas, including the parser, HiveServer2, and cleaning
>> > up unit tests and documentation.  Please join me in welcoming Zoltan to
>> > Hive.
>> >
>> > Zoltan, feel free to say a few words introducing yourself if you
>> would
>> > like to.
>> >
>> > Alan.
>> >
>> >
>> >
>>

[jira] [Created] (HIVE-15915) Emit progress percentage in getting operation status

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-15915:
--

 Summary: Emit progress percentage in getting operation status
 Key: HIVE-15915
 URL: https://issues.apache.org/jira/browse/HIVE-15915
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


When running a query asynchronously, client may want to check the progress 
periodically. HIVE-15473 is to support progressing bar on beeline for Tez. For 
this issue, we just want the progress percentage. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Invitation for Hive committers to become ORC committers

[Jimmy Xiang]

Hi Owen,

I am also interested.

Thanks,
Jimmy


On Sun, Jan 22, 2017 at 2:16 PM, Carl Steinbach  wrote:
> Owen, sorry for the typo.
>
> On Sun, Jan 22, 2017 at 2:16 PM, Carl Steinbach 
> wrote:
>
>> Hi Own,
>>
>> I am interested.
>>
>> Thanks.
>>
>> - Carl
>>
>> On Mon, Jan 9, 2017 at 4:21 PM, Owen O'Malley  wrote:
>>
>>> Ok, I added Sid as well.
>>>
>>> .. Owen
>>>
>>> On Wed, Jan 4, 2017 at 10:01 PM, Siddharth Seth  wrote:
>>>
>>> > Hi Owen,
>>> > I'd be interested as well, if not too late.
>>> > Thanks,
>>> > Sid
>>> >
>>> > On Wed, Jan 4, 2017 at 10:34 AM, Owen O'Malley 
>>> wrote:
>>> >
>>> > > Ferd, I've added you.
>>> > >
>>> > > Suneel, I'm sorry, but the offer is limited to current Hive
>>> committers.
>>> > > http://people.apache.org/phonebook.html?unix=hive
>>> > >
>>> > > .. Owen
>>> > >
>>> > > On Mon, Jan 2, 2017 at 6:39 PM, Suneel Jakka 
>>> > > wrote:
>>> > >
>>> > > > Hi Owen,
>>> > > >
>>> > > > Am also interested.
>>> > > >
>>> > > > Regards,
>>> > > > Suneel Jakka
>>> > > >
>>> > > >
>>> > > > On Mon, Jan 2, 2017 at 8:19 PM, Xu, Cheng A 
>>> > > wrote:
>>> > > >
>>> > > > > Hi Owen,
>>> > > > > Sorry for my late response. I'm also interested.
>>> > > > >
>>> > > > > Thanks,
>>> > > > > Ferd
>>> > > > >
>>> > > > > -Original Message-
>>> > > > > From: Owen O'Malley [mailto:omal...@apache.org]
>>> > > > > Sent: Friday, December 23, 2016 11:55 AM
>>> > > > > To: dev@hive.apache.org
>>> > > > > Subject: Re: Invitation for Hive committers to become ORC
>>> committers
>>> > > > >
>>> > > > > Ok, I believe that I have got everyone. If you don't have karma as
>>> > > shown
>>> > > > > here: http://people.apache.org/phonebook.html?unix=orc
>>> > > > >
>>> > > > > Please, let me know. I believe I have also updated the ORC website
>>> > with
>>> > > > > everyone.
>>> > > > >
>>> > > > > Thanks,
>>> > > > >Owen
>>> > > > >
>>> > > > > On Sat, Dec 17, 2016 at 5:16 AM, Lars Francke <
>>> > lars.fran...@gmail.com>
>>> > > > > wrote:
>>> > > > >
>>> > > > > > Hi Owen,
>>> > > > > >
>>> > > > > > I'm also interested.
>>> > > > > >
>>> > > > > > Thanks,
>>> > > > > > Lars
>>> > > > > >
>>> > > > > > On Fri, Dec 16, 2016 at 10:20 PM, Sergio Pena
>>> > > > > > 
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > Hi Ownen,
>>> > > > > > >
>>> > > > > > > I'm also interested.
>>> > > > > > > - Sergio
>>> > > > > > >
>>> > > > > > > On Fri, Dec 16, 2016 at 11:39 AM, Daniel Dai <
>>> > > da...@hortonworks.com>
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > I am interested.
>>> > > > > > > >
>>> > > > > > > > Thanks,
>>> > > > > > > > Daniel
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On 12/15/16, 1:12 PM, "Owen O'Malley" 
>>> > > wrote:
>>> > > > > > > >
>>> > > > > > > > >All,
>>> > > > > > > > >   As you are aware, we are in the last stages of removing
>>> the
>>> > > > > > > > >forked
>>> > > > > > > ORC
>>> > > > > > > > >code out of Hive. The goal of moving ORC out of Hive was to
>>> > > > > > > > >increase
>>> > > > > > its
>>> > > > > > > > >community and we want to be very deliberately inclusive of
>>> the
>>> > > > > > > > >Hive development community. Towards that end, the ORC PMC
>>> > wants
>>> > > > > > > > >to welcome anyone who is already a Hive committer to
>>> become a
>>> > > > > committer on ORC.
>>> > > > > > > > >
>>> > > > > > > > >  Please respond on this thread to let us know if you are
>>> > > > > interested.
>>> > > > > > > > >
>>> > > > > > > > >Thanks,
>>> > > > > > > > >   Owen on behalf of the ORC PMC
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
>>

Re: [ANNOUNCE] New Hive Committer - Rajesh Balamohan

[Jimmy Xiang]

Congrats, Rajesh!!

On Wed, Dec 14, 2016 at 11:32 AM, Sergey Shelukhin
 wrote:
> Congratulations!
>
> From: Chao Sun 
> Reply-To: "u...@hive.apache.org" 
> Date: Wednesday, December 14, 2016 at 10:52
> To: "dev@hive.apache.org" 
> Cc: "u...@hive.apache.org" , "rbalamo...@apache.org"
> 
> Subject: Re: [ANNOUNCE] New Hive Committer - Rajesh Balamohan
>
> Congrats Rajesh!
>
> On Wed, Dec 14, 2016 at 9:26 AM, Vihang Karajgaonkar 
> wrote:
>>
>> Congrats Rajesh!
>>
>> On Wed, Dec 14, 2016 at 1:54 AM, Jesus Camacho Rodriguez <
>> jcamachorodrig...@hortonworks.com> wrote:
>>
>> > Congrats Rajesh, well deserved! :)
>> >
>> > --
>> > Jesús
>> >
>> >
>> >
>> >
>> > On 12/14/16, 8:41 AM, "Lefty Leverenz"  wrote:
>> >
>> > >Congratulations Rajesh!
>> > >
>> > >-- Lefty
>> > >
>> > >
>> > >On Tue, Dec 13, 2016 at 11:58 PM, Rajesh Balamohan
>> > > > > >
>> > >wrote:
>> > >
>> > >> Thanks a lot for providing this opportunity and to all for their
>> > messages.
>> > >> :)
>> > >>
>> > >> ~Rajesh.B
>> > >>
>> > >> On Wed, Dec 14, 2016 at 11:33 AM, Dharmesh Kakadia
>> > >> > > >
>> > >> wrote:
>> > >>
>> > >> > Congrats Rajesh !
>> > >> >
>> > >> > Thanks,
>> > >> > Dharmesh
>> > >> >
>> > >> > On Tue, Dec 13, 2016 at 7:37 PM, Vikram Dixit K <
>> > vikram.di...@gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> >> Congrats Rajesh! :)
>> > >> >>
>> > >> >> On Tue, Dec 13, 2016 at 9:36 PM, Pengcheng Xiong
>> > >> >> 
>> > >> >> wrote:
>> > >> >>
>> > >> >>> Congrats Rajesh! :)
>> > >> >>>
>> > >> >>> On Tue, Dec 13, 2016 at 6:51 PM, Prasanth Jayachandran <
>> > >> >>> prasan...@apache.org
>> > >> >>> > wrote:
>> > >> >>>
>> > >> >>> > The Apache Hive PMC has voted to make Rajesh Balamohan a
>> > committer on
>> > >> >>> the
>> > >> >>> > Apache Hive Project. Please join me in congratulating Rajesh.
>> > >> >>> >
>> > >> >>> > Congratulations Rajesh!
>> > >> >>> >
>> > >> >>> > Thanks
>> > >> >>> > Prasanth
>> > >> >>>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> --
>> > >> >> Nothing better than when appreciated for hard work.
>> > >> >> -Mark
>> > >> >>
>> > >> >
>> > >> >
>> > >>
>> >
>
>

[jira] [Created] (HIVE-15208) Query string should be HTML encoded for Web UI

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-15208:
--

 Summary: Query string should be HTML encoded for Web UI
 Key: HIVE-15208
 URL: https://issues.apache.org/jira/browse/HIVE-15208
 Project: Hive
  Issue Type: Bug
  Components: Web UI
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14928) Analyze table no scan mess up schema

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-14928:
--

 Summary: Analyze table no scan mess up schema
 Key: HIVE-14928
 URL: https://issues.apache.org/jira/browse/HIVE-14928
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang


StatsNoJobTask uses static variables partUpdates and  table to track stats 
changes. If multiple analyze no scan tasks run at the same time, then 
table/partition schema could mess up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New PMC Member : Pengcheng

[Jimmy Xiang]

Congrats!!

On Mon, Jul 18, 2016 at 9:55 AM, Vihang Karajgaonkar
 wrote:
> Congratulations!
>
>> On Jul 18, 2016, at 5:28 AM, Peter Vary  wrote:
>>
>> Congratulations Pengcheng!
>>
>>
>>> On Jul 18, 2016, at 6:55 AM, Wei Zheng  wrote:
>>>
>>> Congrats Pengcheng!
>>>
>>> Thanks,
>>>
>>> Wei
>>>
>>>
>>>
>>>
>>>
>>>
>>> On 7/17/16, 16:01, "Xuefu Zhang"  wrote:
>>>
 Congrats, PengCheng!

 On Sun, Jul 17, 2016 at 2:28 PM, Sushanth Sowmyan 
 wrote:

> Welcome aboard Pengcheng! :)
>
> On Jul 17, 2016 12:01, "Lefty Leverenz"  wrote:
>
>> Congratulations Pengcheng!
>>
>> -- Lefty
>>
>> On Sun, Jul 17, 2016 at 1:03 PM, Ashutosh Chauhan 
>> wrote:
>>

 Hello Hive community,

 I'm pleased to announce that Pengcheng Xiong has accepted the Apache
>>> Hive
 PMC's
 invitation, and is now our newest PMC member. Many thanks to Pengcheng
>>> for
 all of his hard work.

 Please join me congratulating Pengcheng!

 Best,
 Ashutosh
 (On behalf of the Apache Hive PMC)

>>>
>>
>>
>
>>
>

Re: [ANNOUNCE] New PMC Member : Jesus

[Jimmy Xiang]

Congrats!!

On Mon, Jul 18, 2016 at 9:54 AM, Vihang Karajgaonkar
 wrote:
> Congratulations Jesus!
>
>> On Jul 18, 2016, at 8:30 AM, Sergio Pena  wrote:
>>
>> Congrats Jesus !!!
>>
>> On Mon, Jul 18, 2016 at 7:28 AM, Peter Vary  wrote:
>>
>>> Congratulations Jesus!
>>>
 On Jul 18, 2016, at 6:55 AM, Wei Zheng  wrote:

 Congrats Jesus!

 Thanks,

 Wei







 On 7/17/16, 14:29, "Sushanth Sowmyan"  wrote:

> Good to have you onboard, Jesus! :)
>
> On Jul 17, 2016 12:00, "Lefty Leverenz" 
>>> wrote:
>
>> Congratulations Jesus!
>>
>> -- Lefty
>>
>> On Sun, Jul 17, 2016 at 1:01 PM, Ashutosh Chauhan <
>>> hashut...@apache.org>
>> wrote:
>>
>>> Hello Hive community,
>>>
>>> I'm pleased to announce that Jesus Camacho Rodriguez has accepted the
>>> Apache Hive PMC's
>>> invitation, and is now our newest PMC member. Many thanks to Jesus for
>>> all of
>>> his hard work.
>>>
>>> Please join me congratulating Jesus!
>>>
>>> Best,
>>> Ashutosh
>>> (On behalf of the Apache Hive PMC)
>>>
>>
>>
>>>
>>>
>

Re: [Announce] New Hive Committer - Mohit Sabharwal

[Jimmy Xiang]

Congrats!!

On Fri, Jul 1, 2016 at 1:04 PM, Lenni Kuff  wrote:
> Congrats Mohit!
>
> On Fri, Jul 1, 2016 at 3:27 PM, Peter Vary  wrote:
>
>> Congratulations Mohit!
>> 2016. júl. 1. 19:10 ezt írta ("Vihang Karajgaonkar" > >):
>>
>> > Congratulations Mohit!
>> >
>> > > On Jul 1, 2016, at 10:05 AM, Chao Sun  wrote:
>> > >
>> > > Congratulations Mohit! Good job!
>> > >
>> > > Best,
>> > > Chao
>> > >
>> > > On Fri, Jul 1, 2016 at 9:57 AM, Szehon Ho > > > wrote:
>> > > On behalf of the Apache Hive PMC, I'm pleased to announce that Mohit
>> > Sabharwal has been voted a committer on the Apache Hive project.
>> > >
>> > > Please join me in congratulating Mohit !
>> > >
>> > > Thanks,
>> > > Szehon
>> > >
>> >
>> >
>>

[jira] [Created] (HIVE-13782) Compile async query asynchronously

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-13782:
--

 Summary: Compile async query asynchronously
 Key: HIVE-13782
 URL: https://issues.apache.org/jira/browse/HIVE-13782
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


Currently, when an async query is submitted to HS2, HS2 does the preparation 
synchronously. One of the preparation step is to compile the query, which may 
take some time. It will be helpful to provide an option to do the compilation 
asynchronously.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13679) Pass diagnostic message to failure hooks

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-13679:
--

 Summary: Pass diagnostic message to failure hooks
 Key: HIVE-13679
 URL: https://issues.apache.org/jira/browse/HIVE-13679
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


Pass diagnostic message to failure hooks. This is useful for debugging remote 
job failures.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13559) Pass exception to failure hooks

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-13559:
--

 Summary: Pass exception to failure hooks
 Key: HIVE-13559
 URL: https://issues.apache.org/jira/browse/HIVE-13559
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


Pass exception to failure hooks so that they know more about the failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13501) Invoke failure hooks if query fails on exception

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-13501:
--

 Summary: Invoke failure hooks if query fails on exception
 Key: HIVE-13501
 URL: https://issues.apache.org/jira/browse/HIVE-13501
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


When a query fails on some exception, failure hooks are not called currently. 
It's better to invoke such hooks so that we know the query is failed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13430) Pass error message to failure hook

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-13430:
--

 Summary: Pass error message to failure hook
 Key: HIVE-13430
 URL: https://issues.apache.org/jira/browse/HIVE-13430
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


Currently, the failure hook just knows the query failed. But it has no clue 
what the error is. It is better to pass the error message to the hook.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 45128: HIVE-12619 Switching the field order within an array of structs causes the query to fail

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45128/
---

Review request for hive and Sergio Pena.


Bugs: HIVE-12619
https://issues.apache.org/jira/browse/HIVE-12619


Repository: hive-git


Description
---

This patch makes sure if the order of fields in a struct is changed, the 
projected type should have the right field order so that the query doesn't fail.


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java
 b01f21f 
  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java
 53f3b72 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/ParquetHiveSerDe.java 
e1bf8e2 
  ql/src/test/queries/clientpositive/parquet_schema_evolution.q 193400f 
  ql/src/test/results/clientpositive/parquet_map_null.q.java1.8.out dd541a5 
  ql/src/test/results/clientpositive/parquet_schema_evolution.q.out 0b88d84 
  ql/src/test/results/clientpositive/parquet_type_promotion.q.out 55f9b27 

Diff: https://reviews.apache.org/r/45128/diff/


Testing
---

unit test and test with my local cluster


Thanks,

Jimmy Xiang

Re: [ANNOUNCE] New Hive Committer - Wei Zheng

[Jimmy Xiang]

Congrats Wei!

On Thu, Mar 10, 2016 at 10:55 AM, Thejas Nair  wrote:
> Congrats Wei!
>
>
> On Thu, Mar 10, 2016 at 10:31 AM, Wei Zheng  wrote:
>> Thanks folks! I'm happy to be part of the community and able to contribute!
>>
>> Thanks,
>> Wei
>>
>> From: Eugene Koifman 
>> mailto:ekoif...@hortonworks.com>>
>> Reply-To: "u...@hive.apache.org" 
>> mailto:u...@hive.apache.org>>
>> Date: Thursday, March 10, 2016 at 10:25
>> To: "u...@hive.apache.org" 
>> mailto:u...@hive.apache.org>>
>> Cc: "dev@hive.apache.org" 
>> mailto:dev@hive.apache.org>>, 
>> "w...@apache.org" 
>> mailto:w...@apache.org>>
>> Subject: Re: [ANNOUNCE] New Hive Committer - Wei Zheng
>>
>> Congratulations!
>>
>> From: Hari Sivarama Subramaniyan 
>> mailto:hsubramani...@hortonworks.com>>
>> Reply-To: "u...@hive.apache.org" 
>> mailto:u...@hive.apache.org>>
>> Date: Thursday, March 10, 2016 at 10:02 AM
>> To: "u...@hive.apache.org" 
>> mailto:u...@hive.apache.org>>
>> Cc: "dev@hive.apache.org" 
>> mailto:dev@hive.apache.org>>, 
>> "w...@apache.org" 
>> mailto:w...@apache.org>>
>> Subject: Re: [ANNOUNCE] New Hive Committer - Wei Zheng
>>
>>
>> Congrats Wei.
>>
>>
>> Thanks
>>
>> Hari
>>
>> 
>> From: Pengcheng Xiong mailto:pxi...@apache.org>>
>> Sent: Thursday, March 10, 2016 9:50 AM
>> To: u...@hive.apache.org
>> Cc: dev@hive.apache.org; 
>> w...@apache.org
>> Subject: Re: [ANNOUNCE] New Hive Committer - Wei Zheng
>>
>> Big congrats Wei!
>>
>> Pengcheng
>>
>> On Thu, Mar 10, 2016 at 9:25 AM, Vijay K.N 
>> mailto:vijayja...@gmail.com>> wrote:
>>
>> Congrats Wei Zheng!!
>>
>> On Mar 10, 2016 6:57 AM, "Vikram Dixit K" 
>> mailto:vik...@apache.org>> wrote:
>> The Apache Hive PMC has voted to make Wei Zheng a committer on the Apache 
>> Hive Project. Please join me in congratulating Wei.
>>
>> Thanks
>> Vikram.
>>

[jira] [Created] (HIVE-13237) Select parquet struct field with upper case throws NPE

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-13237:
--

 Summary: Select parquet struct field with upper case throws NPE
 Key: HIVE-13237
 URL: https://issues.apache.org/jira/browse/HIVE-13237
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


Query "select msg.fieldone from test" throws NPE if msg's fieldone is actually 
fieldOne:

{noformat}
2016-03-08 17:30:57,772 ERROR [main]: exec.FetchTask 
(FetchTask.java:initialize(86)) - java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.ExprNodeFieldEvaluator.initialize(ExprNodeFieldEvaluator.java:61)
at 
org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)
at 
org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:980)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:63)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13043) Reload function has no impact to function registry

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-13043:
--

 Summary: Reload function has no impact to function registry
 Key: HIVE-13043
 URL: https://issues.apache.org/jira/browse/HIVE-13043
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


With HIVE-2573, users should run "reload function" to refresh cached function 
registry. However, "reload function" has no impact at all. We need to fix this. 
Otherwise, HS2 needs to be restarted to see new global functions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-13026) Pending/running operation metrics are wrong

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-13026:
--

 Summary: Pending/running operation metrics are wrong
 Key: HIVE-13026
 URL: https://issues.apache.org/jira/browse/HIVE-13026
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


A query is finished, however the pending/running operation count doesn't 
decrease. 

For example, in TestHs2Metrics::testMetrics(), we have
{noformat}
MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.TIMER, 
"api_hs2_operation_PENDING", 1);
MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.TIMER, 
"api_hs2_operation_RUNNING", 1);
MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.COUNTER, 
"hs2_completed_operation_FINISHED", 1);
{noformat}

Should it be below?

{noformat}
MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.TIMER, 
"api_hs2_operation_PENDING", 0);
MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.TIMER, 
"api_hs2_operation_RUNNING", 0);
MetricsTestUtils.verifyMetricsJson(json, MetricsTestUtils.COUNTER, 
"hs2_completed_operation_FINISHED", 1);
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12987) Add metrics for HS2 active users and SQL operations

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12987:
--

 Summary: Add metrics for HS2 active users and SQL operations
 Key: HIVE-12987
 URL: https://issues.apache.org/jira/browse/HIVE-12987
 Project: Hive
  Issue Type: Task
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


HIVE-12271 added metrics for all HS2 operations. Sometimes, users are also 
interested in metrics just for SQL operations.

It is useful to track active user count as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41062: HIVE-12485 Secure HS2 web UI with kerberos

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41062/
---

(Updated Dec. 9, 2015, 4:35 p.m.)


Review request for hive, Szehon Ho and Xuefu Zhang.


Changes
---

Changed use.SSL/SPNEGO to lower case.


Bugs: HIVE-12485
https://issues.apache.org/jira/browse/HIVE-12485


Repository: hive-git


Description
---

Added an AuthenticationFilter to secure the HS2 web ui with kerberos


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994 
  common/src/java/org/apache/hive/http/HttpServer.java 4b0ed68 
  service/src/java/org/apache/hive/service/server/HiveServer2.java cad541a 

Diff: https://reviews.apache.org/r/41062/diff/


Testing
---

Manually tested it locally.


Thanks,

Jimmy Xiang

Re: Review Request 41062: HIVE-12485 Secure HS2 web UI with kerberos

[Jimmy Xiang]

> On Dec. 8, 2015, 10:10 p.m., Mohit Sabharwal wrote:
> > common/src/java/org/apache/hive/http/HttpServer.java, line 93
> > <https://reviews.apache.org/r/41062/diff/1/?file=1155466#file1155466line93>
> >
> > should this be conditional on the cluster itself being kerberized ?
> > 
> > i.e.
> > if (UserGroupInformation.isSecurityEnabled()) {
> > }

No need. When I do the testing, my cluster is not secured.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41062/#review109411
-------


On Dec. 8, 2015, 12:05 a.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41062/
> ---
> 
> (Updated Dec. 8, 2015, 12:05 a.m.)
> 
> 
> Review request for hive, Szehon Ho and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12485
> https://issues.apache.org/jira/browse/HIVE-12485
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added an AuthenticationFilter to secure the HS2 web ui with kerberos
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994 
>   common/src/java/org/apache/hive/http/HttpServer.java 4b0ed68 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java cad541a 
> 
> Diff: https://reviews.apache.org/r/41062/diff/
> 
> 
> Testing
> ---
> 
> Manually tested it locally.
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 41062: HIVE-12485 Secure HS2 web UI with kerberos

[Jimmy Xiang]

> On Dec. 8, 2015, 10:43 p.m., Szehon Ho wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 1861
> > <https://reviews.apache.org/r/41062/diff/1/?file=1155465#file1155465line1861>
> >
> > We can keep it lower-case to keep it consistent.

I thought about this too. We have use.SSL and ssl in other configuration 
parameters. I am open to change it to lower-case.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41062/#review109419
---


On Dec. 8, 2015, 12:05 a.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41062/
> ---
> 
> (Updated Dec. 8, 2015, 12:05 a.m.)
> 
> 
> Review request for hive, Szehon Ho and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12485
> https://issues.apache.org/jira/browse/HIVE-12485
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added an AuthenticationFilter to secure the HS2 web ui with kerberos
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994 
>   common/src/java/org/apache/hive/http/HttpServer.java 4b0ed68 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java cad541a 
> 
> Diff: https://reviews.apache.org/r/41062/diff/
> 
> 
> Testing
> ---
> 
> Manually tested it locally.
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Review Request 41062: HIVE-12485 Secure HS2 web UI with kerberos

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41062/
---

Review request for hive, Szehon Ho and Xuefu Zhang.


Bugs: HIVE-12485
https://issues.apache.org/jira/browse/HIVE-12485


Repository: hive-git


Description
---

Added an AuthenticationFilter to secure the HS2 web ui with kerberos


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d52f994 
  common/src/java/org/apache/hive/http/HttpServer.java 4b0ed68 
  service/src/java/org/apache/hive/service/server/HiveServer2.java cad541a 

Diff: https://reviews.apache.org/r/41062/diff/


Testing
---

Manually tested it locally.


Thanks,

Jimmy Xiang

Re: Review Request 40833: HIVE-12471 Secure HS2 web UI with SSL

[Jimmy Xiang]

> On Dec. 2, 2015, 9:23 p.m., Mohit Sabharwal wrote:
> > The conditional is reverse, otherwise lgtm

Good catch! Will fix. Thanks.


> On Dec. 2, 2015, 9:23 p.m., Mohit Sabharwal wrote:
> > common/src/java/org/apache/hive/http/HttpServer.java, line 286
> > <https://reviews.apache.org/r/40833/diff/2/?file=1151660#file1151660line286>
> >
> > (!b.useSSL) ?

Fixed.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40833/#review108725
---


On Dec. 2, 2015, 8:09 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40833/
> ---
> 
> (Updated Dec. 2, 2015, 8:09 p.m.)
> 
> 
> Review request for hive, Lefty Leverenz, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12471
> https://issues.apache.org/jira/browse/HIVE-12471
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch changed the HttpServer construction to use builder which is 
> flexible. It addes 3 configurations for web UI SSL support. By default, it is 
> disabled.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f96071 
>   common/src/java/org/apache/hive/http/HttpServer.java 1ff8d7c 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 204eb5a 
> 
> Diff: https://reviews.apache.org/r/40833/diff/
> 
> 
> Testing
> ---
> 
> Tested web UI with and without SSL, both work as expected.
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 40833: HIVE-12471 Secure HS2 web UI with SSL

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40833/
---

(Updated Dec. 2, 2015, 8:09 p.m.)


Review request for hive, Lefty Leverenz, Szehon Ho, and Xuefu Zhang.


Bugs: HIVE-12471
https://issues.apache.org/jira/browse/HIVE-12471


Repository: hive-git


Description
---

This patch changed the HttpServer construction to use builder which is 
flexible. It addes 3 configurations for web UI SSL support. By default, it is 
disabled.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f96071 
  common/src/java/org/apache/hive/http/HttpServer.java 1ff8d7c 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 204eb5a 

Diff: https://reviews.apache.org/r/40833/diff/


Testing
---

Tested web UI with and without SSL, both work as expected.


Thanks,

Jimmy Xiang

Re: Review Request 40833: HIVE-12471 Secure HS2 web UI with SSL

[Jimmy Xiang]

> On Dec. 2, 2015, 6:41 a.m., Mohit Sabharwal wrote:
> > common/src/java/org/apache/hive/http/HttpServer.java, line 280
> > <https://reviews.apache.org/r/40833/diff/1/?file=1149773#file1149773line280>
> >
> > Nit: b.keyStorePath to determine ssl is not enabled looks bit unclean. 
> > Maybe add a useSsl boolean to the builder ?

Ok.


> On Dec. 2, 2015, 6:41 a.m., Mohit Sabharwal wrote:
> > common/src/java/org/apache/hive/http/HttpServer.java, line 311
> > <https://reviews.apache.org/r/40833/diff/1/?file=1149773#file1149773line311>
> >
> > Any reason why it shouldn't just default to what's specified in 
> > HiveConf.HIVE_SERVER2_WEBUI_MAX_THREADS (which is 50) ?

It is because this HttpServer can be used for HMS web UI too. I changed the 
logic a little so that we don't hardcode the number here.


> On Dec. 2, 2015, 6:41 a.m., Mohit Sabharwal wrote:
> > common/src/java/org/apache/hive/http/HttpServer.java, line 302
> > <https://reviews.apache.org/r/40833/diff/1/?file=1149773#file1149773line302>
> >
> > Add back the comment 
> > 
> > /* Set servlet context attribute that can be used in jsp */
> > 
> > It wasn't clear to me this is related to jsp

Added it to setContextAttributes method. You can take a look the jsp file to 
see how it is used. Basically, this is a way to pass information to jsp page.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40833/#review108613
---


On Dec. 1, 2015, 6:50 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40833/
> ---
> 
> (Updated Dec. 1, 2015, 6:50 p.m.)
> 
> 
> Review request for hive, Lefty Leverenz, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12471
> https://issues.apache.org/jira/browse/HIVE-12471
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch changed the HttpServer construction to use builder which is 
> flexible. It addes 3 configurations for web UI SSL support. By default, it is 
> disabled.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9e805bd 
>   common/src/java/org/apache/hive/http/HttpServer.java 1ff8d7c 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 204eb5a 
> 
> Diff: https://reviews.apache.org/r/40833/diff/
> 
> 
> Testing
> ---
> 
> Tested web UI with and without SSL, both work as expected.
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 40785: HIVE-12506: SHOW CREATE TABLE command creates a table that does not work for RCFile format

[Jimmy Xiang]

> On Dec. 1, 2015, 10:45 p.m., Jimmy Xiang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java, line 2128
> > <https://reviews.apache.org/r/40785/diff/1/?file=1148109#file1148109line2128>
> >
> > If this is the only statement in the block, can we merge two if 
> > statements into one?
> 
> Chaoyu Tang wrote:
> Jimmy, could you clarify? this is the code with patch:
> ---
>   if (tbl.getStorageHandler() == null) {
> // If serialization.format property has the default value, it 
> will not to be included in
> // SERDE properties
> if 
> (MetaStoreUtils.DEFAULT_SERIALIZATION_FORMAT.equals(serdeParams.get(
> serdeConstants.SERIALIZATION_FORMAT))){
>   serdeParams.remove(serdeConstants.SERIALIZATION_FORMAT);
> }
> if (!serdeParams.isEmpty()) {
>   appendSerdeParams(tbl_row_format, serdeParams).append(" \n");
> }
> tbl_row_format.append("STORED AS INPUTFORMAT \n  '" +
> escapeHiveCommand(sd.getInputFormat()) + "' \n");
> tbl_row_format.append("OUTPUTFORMAT \n  '" +
> escapeHiveCommand(sd.getOutputFormat()) + "'");
>   } else {
>   ...
> ---
> You meant which two if can be consolidated?

I see. There is a tab so I thought there is just one if statement in the block 
by mistake.


> On Dec. 1, 2015, 10:45 p.m., Jimmy Xiang wrote:
> > ql/src/test/results/clientpositive/show_create_table_delimited.q.out, line 
> > 36
> > <https://reviews.apache.org/r/40785/diff/1/?file=1148112#file1148112line36>
> >
> > Now we expose those serde properties names. This is ok, right?
> 
> Chaoyu Tang wrote:
> I think it should be fine since other Hive commands like desc extended 
> tblName already expose these serde property names in their output. for 
> example, here is partial output of desc extended tblName:
> serdeInfo:SerDeInfo(name:null, 
> serializationLib:org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe, 
> parameters:{colelction.delim=,, mapkey.delim=:, serialization.format=|, 
> field.delim=|}),

ok


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40785/#review108573
---


On Nov. 29, 2015, 5:47 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40785/
> ---
> 
> (Updated Nov. 29, 2015, 5:47 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> More than one SerDe in Hive (e.g. LazySimpleSerDe, ColumnarSerDe) use the 
> DELIMITED properties, so "show create table" output with DELIMTED instead of 
> SERDE [WITH SERDEPROPERTIES] is not sufficient to be used to recreate the 
> same table.
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 02cbd76 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
>   ql/src/test/results/clientpositive/nullformat.q.out 6cfc2b8 
>   ql/src/test/results/clientpositive/nullformatCTAS.q.out 7adca68 
>   ql/src/test/results/clientpositive/show_create_table_delimited.q.out 
> 79f29de 
> 
> Diff: https://reviews.apache.org/r/40785/diff/
> 
> 
> Testing
> ---
> 
> 1. Manual tests passed
> 2. has been submitted for pre-commit build.
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 40785: HIVE-12506: SHOW CREATE TABLE command creates a table that does not work for RCFile format

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40785/#review108582
---

Ship it!


Ship It!

- Jimmy Xiang


On Nov. 29, 2015, 5:47 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40785/
> ---
> 
> (Updated Nov. 29, 2015, 5:47 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> More than one SerDe in Hive (e.g. LazySimpleSerDe, ColumnarSerDe) use the 
> DELIMITED properties, so "show create table" output with DELIMTED instead of 
> SERDE [WITH SERDEPROPERTIES] is not sufficient to be used to recreate the 
> same table.
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 02cbd76 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
>   ql/src/test/results/clientpositive/nullformat.q.out 6cfc2b8 
>   ql/src/test/results/clientpositive/nullformatCTAS.q.out 7adca68 
>   ql/src/test/results/clientpositive/show_create_table_delimited.q.out 
> 79f29de 
> 
> Diff: https://reviews.apache.org/r/40785/diff/
> 
> 
> Testing
> ---
> 
> 1. Manual tests passed
> 2. has been submitted for pre-commit build.
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 40785: HIVE-12506: SHOW CREATE TABLE command creates a table that does not work for RCFile format

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40785/#review108573
---


Looks ok to me. Just some questions.


ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java (line 2122)
<https://reviews.apache.org/r/40785/#comment168024>

So now we always show these params? Do we have some default value for each 
paramter and skip those properties with the default value?



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java (line 2126)
<https://reviews.apache.org/r/40785/#comment168023>

If this is the only statement in the block, can we merge two if statements 
into one?



ql/src/test/results/clientpositive/show_create_table_delimited.q.out (line 36)
<https://reviews.apache.org/r/40785/#comment168025>

Now we expose those serde properties names. This is ok, right?


- Jimmy Xiang


On Nov. 29, 2015, 5:47 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40785/
> ---
> 
> (Updated Nov. 29, 2015, 5:47 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> More than one SerDe in Hive (e.g. LazySimpleSerDe, ColumnarSerDe) use the 
> DELIMITED properties, so "show create table" output with DELIMTED instead of 
> SERDE [WITH SERDEPROPERTIES] is not sufficient to be used to recreate the 
> same table.
> 
> 
> Diffs
> -
> 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 02cbd76 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a210b95 
>   ql/src/test/results/clientpositive/nullformat.q.out 6cfc2b8 
>   ql/src/test/results/clientpositive/nullformatCTAS.q.out 7adca68 
>   ql/src/test/results/clientpositive/show_create_table_delimited.q.out 
> 79f29de 
> 
> Diff: https://reviews.apache.org/r/40785/diff/
> 
> 
> Testing
> ---
> 
> 1. Manual tests passed
> 2. has been submitted for pre-commit build.
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Review Request 40833: HIVE-12471 Secure HS2 web UI with SSL

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40833/
---

Review request for hive, Lefty Leverenz, Szehon Ho, and Xuefu Zhang.


Repository: hive-git


Description
---

This patch changed the HttpServer construction to use builder which is 
flexible. It addes 3 configurations for web UI SSL support. By default, it is 
disabled.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9e805bd 
  common/src/java/org/apache/hive/http/HttpServer.java 1ff8d7c 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 204eb5a 

Diff: https://reviews.apache.org/r/40833/diff/


Testing
---

Tested web UI with and without SSL, both work as expected.


Thanks,

Jimmy Xiang

Re: Review Request 40833: HIVE-12471 Secure HS2 web UI with SSL

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40833/
---

(Updated Dec. 1, 2015, 6:50 p.m.)


Review request for hive, Lefty Leverenz, Szehon Ho, and Xuefu Zhang.


Bugs: HIVE-12471
https://issues.apache.org/jira/browse/HIVE-12471


Repository: hive-git


Description
---

This patch changed the HttpServer construction to use builder which is 
flexible. It addes 3 configurations for web UI SSL support. By default, it is 
disabled.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 9e805bd 
  common/src/java/org/apache/hive/http/HttpServer.java 1ff8d7c 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 204eb5a 

Diff: https://reviews.apache.org/r/40833/diff/


Testing
---

Tested web UI with and without SSL, both work as expected.


Thanks,

Jimmy Xiang

Re: [ANNOUNCE] New PMC Member : John Pullokkaran

[Jimmy Xiang]

Congrats!!

On Tue, Nov 24, 2015 at 3:04 PM, Szehon Ho  wrote:

> Congratulations!
>
> On Tue, Nov 24, 2015 at 3:02 PM, Xuefu Zhang  wrote:
>
> > Congratulations, John!
> >
> > --Xuefu
> >
> > On Tue, Nov 24, 2015 at 3:01 PM, Prasanth J 
> > wrote:
> >
> >> Congratulations and Welcome John!
> >>
> >> Thanks
> >> Prasanth
> >>
> >> On Nov 24, 2015, at 4:59 PM, Ashutosh Chauhan 
> >> wrote:
> >>
> >> On behalf of the Hive PMC I am delighted to announce John Pullokkaran is
> >> joining Hive PMC.
> >> John is a long time contributor in Hive and is focusing on compiler and
> >> optimizer areas these days.
> >> Please give John a warm welcome to the project!
> >>
> >> Ashutosh
> >>
> >>
> >>
> >
>

[jira] [Created] (HIVE-12511) IN clause performs differently then = clause

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12511:
--

 Summary: IN clause performs differently then = clause
 Key: HIVE-12511
 URL: https://issues.apache.org/jira/browse/HIVE-12511
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


Similar to HIVE-11973, IN clause performs differently then = clause for "int" 
type with string values.
For example,
{noformat}
SELECT * FROM inttest WHERE iValue IN ('01');
{noformat}
will not return any rows with int iValue = 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40500: HIVE-12338 Add webui to HiveServer2

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40500/
---

(Updated Nov. 24, 2015, 7:36 p.m.)


Review request for hive, Szehon Ho and Xuefu Zhang.


Bugs: HIVE-12338
https://issues.apache.org/jira/browse/HIVE-12338


Repository: hive-git


Description
---

Added web UI to HS2. The UI is similar to those for other Hadoop components.
The default web UI port is set to 10002, which is configurable. It can be 
disabled. Currently it shows active sessions and queries. It can also access 
locals, metrics, and configuration.


Diffs (updated)
-

  common/pom.xml cd14581 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java fffedd9 
  common/src/java/org/apache/hive/http/AdminAuthorizedServlet.java PRE-CREATION 
  common/src/java/org/apache/hive/http/ConfServlet.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java PRE-CREATION 
  common/src/java/org/apache/hive/http/JMXJsonServlet.java PRE-CREATION 
  pom.xml c38c10f 
  ql/pom.xml d893099 
  service/pom.xml afa52cf 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 25cefc2 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
b0bd351 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
1331a99 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 
1ab5652 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
a14908b 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
d11cf3d 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b30b6a2 
  service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp PRE-CREATION 
  service/src/resources/hive-webapps/hiveserver2/index.html PRE-CREATION 
  service/src/resources/hive-webapps/static/css/bootstrap-theme.min.css 
PRE-CREATION 
  service/src/resources/hive-webapps/static/css/bootstrap.min.css PRE-CREATION 
  service/src/resources/hive-webapps/static/css/hive.css PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.eot
 PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.svg
 PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.ttf
 PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.woff
 PRE-CREATION 
  service/src/resources/hive-webapps/static/hive_logo.jpeg PRE-CREATION 
  spark-client/pom.xml 9d2b418 

Diff: https://reviews.apache.org/r/40500/diff/


Testing
---


Thanks,

Jimmy Xiang

Re: Review Request 40500: HIVE-12338 Add webui to HiveServer2

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40500/
---

(Updated Nov. 23, 2015, 9:51 p.m.)


Review request for hive, Szehon Ho and Xuefu Zhang.


Bugs: HIVE-12338
https://issues.apache.org/jira/browse/HIVE-12338


Repository: hive-git


Description
---

Added web UI to HS2. The UI is similar to those for other Hadoop components.
The default web UI port is set to 10002, which is configurable. It can be 
disabled. Currently it shows active sessions and queries. It can also access 
locals, metrics, and configuration.


Diffs (updated)
-

  common/pom.xml cd14581 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java f48403b 
  common/src/java/org/apache/hive/http/AdminAuthorizedServlet.java PRE-CREATION 
  common/src/java/org/apache/hive/http/ConfServlet.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java PRE-CREATION 
  common/src/java/org/apache/hive/http/JMXJsonServlet.java PRE-CREATION 
  pom.xml c6df4a5 
  ql/pom.xml 9420a62 
  service/pom.xml afa52cf 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java 25cefc2 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
b0bd351 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
1331a99 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 
1ab5652 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
a14908b 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
d11cf3d 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b30b6a2 
  service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp PRE-CREATION 
  service/src/resources/hive-webapps/hiveserver2/index.html PRE-CREATION 
  service/src/resources/hive-webapps/static/css/bootstrap-theme.min.css 
PRE-CREATION 
  service/src/resources/hive-webapps/static/css/bootstrap.min.css PRE-CREATION 
  service/src/resources/hive-webapps/static/css/hive.css PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.eot
 PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.svg
 PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.ttf
 PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.woff
 PRE-CREATION 
  service/src/resources/hive-webapps/static/hive_logo.jpeg PRE-CREATION 
  spark-client/pom.xml a0bbe56 

Diff: https://reviews.apache.org/r/40500/diff/


Testing
---


Thanks,

Jimmy Xiang

Re: Review Request 40500: HIVE-12338 Add webui to HiveServer2

[Jimmy Xiang]

> On Nov. 23, 2015, 9:37 p.m., Szehon Ho wrote:
> > service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp, line 168
> > <https://reviews.apache.org/r/40500/diff/1/?file=1133042#file1133042line168>
> >
> > One more comment, no need to put 'version', just 'When Hive version was 
> > compiled and by whom'

Already fixed in v2. Let me upload it here.


> On Nov. 23, 2015, 9:37 p.m., Szehon Ho wrote:
> > service/src/java/org/apache/hive/service/cli/operation/OperationManager.java,
> >  line 66
> > <https://reviews.apache.org/r/40500/diff/1/?file=1133036#file1133036line66>
> >
> > What do you think about having jsp code filter out operation that are 
> > not SQLOperation?
> > 
> > Seems cleaner than maintaining a separate list in here.

We can address this in HIVE-12484. The other option is to show all operations 
instead of just SQL queries. We can discuss it there.


> On Nov. 23, 2015, 9:37 p.m., Szehon Ho wrote:
> > common/pom.xml, line 61
> > <https://reviews.apache.org/r/40500/diff/1/?file=1133027#file1133027line61>
> >
> > Indent

Cool. Will fix it.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40500/#review107271
---


On Nov. 19, 2015, 8:53 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40500/
> ---
> 
> (Updated Nov. 19, 2015, 8:53 p.m.)
> 
> 
> Review request for hive, Szehon Ho and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12338
> https://issues.apache.org/jira/browse/HIVE-12338
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added web UI to HS2. The UI is similar to those for other Hadoop components.
> The default web UI port is set to 10002, which is configurable. It can be 
> disabled. Currently it shows active sessions and queries. It can also access 
> locals, metrics, and configuration.
> 
> 
> Diffs
> -
> 
>   common/pom.xml cd14581 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2185f85 
>   common/src/java/org/apache/hive/http/AdminAuthorizedServlet.java 
> PRE-CREATION 
>   common/src/java/org/apache/hive/http/ConfServlet.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/HttpServer.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/JMXJsonServlet.java PRE-CREATION 
>   pom.xml c6df4a5 
>   service/pom.xml afa52cf 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d13415e 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 8b42265 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 
> 1ab5652 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 2d784f0 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> d11cf3d 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b30b6a2 
>   service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp PRE-CREATION 
>   service/src/resources/hive-webapps/hiveserver2/index.html PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/bootstrap-theme.min.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/bootstrap.min.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/hive.css PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.eot
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.svg
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.ttf
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.woff
>  PRE-CREATION 
>   service/src/resources/hive-webapps/static/hive_logo.jpeg PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40500/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 40500: HIVE-12338 Add webui to HiveServer2

[Jimmy Xiang]

> On Nov. 21, 2015, 5:11 a.m., Lefty Leverenz wrote:
> >

Good catches. Thanks a lot for the review.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40500/#review107474
---


On Nov. 19, 2015, 8:53 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40500/
> ---
> 
> (Updated Nov. 19, 2015, 8:53 p.m.)
> 
> 
> Review request for hive, Szehon Ho and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12338
> https://issues.apache.org/jira/browse/HIVE-12338
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added web UI to HS2. The UI is similar to those for other Hadoop components.
> The default web UI port is set to 10002, which is configurable. It can be 
> disabled. Currently it shows active sessions and queries. It can also access 
> locals, metrics, and configuration.
> 
> 
> Diffs
> -
> 
>   common/pom.xml cd14581 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2185f85 
>   common/src/java/org/apache/hive/http/AdminAuthorizedServlet.java 
> PRE-CREATION 
>   common/src/java/org/apache/hive/http/ConfServlet.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/HttpServer.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/JMXJsonServlet.java PRE-CREATION 
>   pom.xml c6df4a5 
>   service/pom.xml afa52cf 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d13415e 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 8b42265 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 
> 1ab5652 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 2d784f0 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> d11cf3d 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b30b6a2 
>   service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp PRE-CREATION 
>   service/src/resources/hive-webapps/hiveserver2/index.html PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/bootstrap-theme.min.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/bootstrap.min.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/hive.css PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.eot
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.svg
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.ttf
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.woff
>  PRE-CREATION 
>   service/src/resources/hive-webapps/static/hive_logo.jpeg PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40500/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: file too large - itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz

[Jimmy Xiang]

I can get the latest of master branch, clean it up, and force-push to
master-fixed again.

Another solution is to abandon branch master, then use a new branch like
"trunk".

Thanks,
Jimmy

On Fri, Nov 20, 2015 at 12:07 PM, Sergey Shelukhin 
wrote:

> Did this ever go anywhere?
> Infra told us to try ourselves.
> I wonder if we should do it before releasing 2.0...
>
> I am assuming master-fixed diverged by now. It can be rebased as follows:
>
>  git rebase --onto master-fixed `git log master --grep="$(git show -s
> master-fixed --format="%s")" --format="%H"` master
>
>  git push origin master:master-fixed (!!!)
>
> Or with safety checks:
> * OLD_HEAD=`git log master --grep="$(git show -s master-fixed
> --format="%s")" --format="%H"`
> * Optional git show $OLD_HEAD and git show master-fixed to see if they
> match
> * git rebase --onto master-fixed `git log master --grep="$(git show -s
> master-fixed --format="%s")" --format="%H"` master
>   This will update the branch correctly but change master, not
> master-fixed (because essentially you are rebasing new commits added to
> master on top of master-fixed)… If someone knows a better way to do it let
> me know.
> * Optionally git diff master origin/master to check.
> * git push origin master:master-fixed.
> * For good measure, git reset --hard origin/master.
>
>
>
>
>
> On 15/11/6, 13:36, "Owen O'Malley"  wrote:
>
> >Ok, I've filled the infra jira for replacing the master branch:
> >
> > https://issues.apache.org/jira/browse/INFRA-10731
> >
> >.. Owen
> >
> >On Fri, Nov 6, 2015 at 9:36 AM, Jimmy Xiang  wrote:
> >
> >> Done. Branch master-fixed is clean now. What should we do next?
> >>
> >> On Fri, Nov 6, 2015 at 9:26 AM, Jimmy Xiang 
> wrote:
> >>
> >> > Let me do that.
> >> >
> >> > On Fri, Nov 6, 2015 at 9:24 AM, Owen O'Malley 
> >> wrote:
> >> >
> >> >> Can someone push a fixed master history to "master-fixed"? Then we
> >>can
> >> >> start a discussion with infra about replacing master with
> >>master-fixed.
> >> >>
> >> >> .. Owen
> >> >>
> >> >> On Fri, Nov 6, 2015 at 9:04 AM, Jimmy Xiang 
> >> wrote:
> >> >>
> >> >> > Right. The file is not in source control any more. But it was
> >>there at
> >> >> some
> >> >> > moment. That's why it is in the git history. So we need to clean up
> >> the
> >> >> git
> >> >> > history.
> >> >> >
> >> >> > To push to a branch other than master, for example, hive-11890, as
> >>a
> >> >> > work-around, you can try
> https://rtyley.github.io/bfg-repo-cleaner/
> >> to
> >> >> do
> >> >> > the clean up first, then push.
> >> >> >
> >> >> > On Fri, Nov 6, 2015 at 9:00 AM, Jimmy Xiang 
> >> >> wrote:
> >> >> >
> >> >> > > I ran into the same problem too. What we need to do is to clean
> >>up
> >> the
> >> >> > git
> >> >> > > history. However, the master branch can't be force-pushed. I was
> >> >> > wondering
> >> >> > > if this is something the infra team can help.
> >> >> > >
> >> >> > > On Fri, Nov 6, 2015 at 8:54 AM, Xuefu Zhang  >
> >> >> wrote:
> >> >> > >
> >> >> > >> I don't quite follow. That file is not in source control, but
> >>only
> >> >> > >> downloaded when running test. Why a git push will mass with that
> >> >> file?
> >> >> > >>
> >> >> > >>
> >> >> > >> On Fri, Nov 6, 2015 at 7:44 AM, Owen O'Malley
> >>
> >> >> > wrote:
> >> >> > >>
> >> >> > >> > Github apparently has a limit on pushes to personal accounts
> >>that
> >> >> no
> >> >> > >> file
> >> >> > >> > may be larger than 100mb. When I try to push a branch to my
> >> >> personal
> >> >> > >> clone
> >> >> > >> > of hive, I get:
> >> >> > >> >
> >> >> > >> &g

[jira] [Created] (HIVE-12485) Secure HS2 web UI with kerberos

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12485:
--

 Summary: Secure HS2 web UI with kerberos
 Key: HIVE-12485
 URL: https://issues.apache.org/jira/browse/HIVE-12485
 Project: Hive
  Issue Type: Sub-task
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12484) Show meta operations on HS2 web UI

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12484:
--

 Summary: Show meta operations on HS2 web UI
 Key: HIVE-12484
 URL: https://issues.apache.org/jira/browse/HIVE-12484
 Project: Hive
  Issue Type: Sub-task
Reporter: Jimmy Xiang


As Mohit pointed out in the review of HIVE-12338, it is nice to show meta 
operations on HS2 web UI too. So that we can have an end-to-end picture for 
those operations access HMS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 40500: HIVE-12338 Add webui to HiveServer2

[Jimmy Xiang]

> On Nov. 20, 2015, 4:53 a.m., Mohit Sabharwal wrote:
> > LGTM. This patch only includes SQLOperations. Are we planning to add 
> > metadata operations as well ? (so we can capture jdbc clients and Hue usage 
> > as well...)
> 
> Jimmy Xiang wrote:
> Thanks a lot for the review. I was thinking about showing meta operations 
> in HMS web UI. Should we also show them here?
> 
> Mohit Sabharwal wrote:
> I guess at some point it'd be good to have it in HS2 UI as well so we can 
> get the end-to-end picture (That way we can show all running operations for a 
> given session, not just the queries). We could do this as an enhancement at a 
> later point.

Right. It is nice to have to get an end-to-end picture. Let's do it as a 
followup enhancement.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40500/#review107283
-------


On Nov. 19, 2015, 8:53 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40500/
> ---
> 
> (Updated Nov. 19, 2015, 8:53 p.m.)
> 
> 
> Review request for hive, Szehon Ho and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12338
> https://issues.apache.org/jira/browse/HIVE-12338
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added web UI to HS2. The UI is similar to those for other Hadoop components.
> The default web UI port is set to 10002, which is configurable. It can be 
> disabled. Currently it shows active sessions and queries. It can also access 
> locals, metrics, and configuration.
> 
> 
> Diffs
> -
> 
>   common/pom.xml cd14581 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2185f85 
>   common/src/java/org/apache/hive/http/AdminAuthorizedServlet.java 
> PRE-CREATION 
>   common/src/java/org/apache/hive/http/ConfServlet.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/HttpServer.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/JMXJsonServlet.java PRE-CREATION 
>   pom.xml c6df4a5 
>   service/pom.xml afa52cf 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d13415e 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 8b42265 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 
> 1ab5652 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 2d784f0 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> d11cf3d 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b30b6a2 
>   service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp PRE-CREATION 
>   service/src/resources/hive-webapps/hiveserver2/index.html PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/bootstrap-theme.min.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/bootstrap.min.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/hive.css PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.eot
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.svg
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.ttf
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.woff
>  PRE-CREATION 
>   service/src/resources/hive-webapps/static/hive_logo.jpeg PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40500/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 40500: HIVE-12338 Add webui to HiveServer2

[Jimmy Xiang]

> On Nov. 20, 2015, 4:53 a.m., Mohit Sabharwal wrote:
> > LGTM. This patch only includes SQLOperations. Are we planning to add 
> > metadata operations as well ? (so we can capture jdbc clients and Hue usage 
> > as well...)

Thanks a lot for the review. I was thinking about showing meta operations in 
HMS web UI. Should we also show them here?


> On Nov. 20, 2015, 4:53 a.m., Mohit Sabharwal wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 1856
> > <https://reviews.apache.org/r/40500/diff/1/?file=1133028#file1133028line1856>
> >
> > Th -> The

Good catch. Fixed.


> On Nov. 20, 2015, 4:53 a.m., Mohit Sabharwal wrote:
> > service/src/java/org/apache/hive/service/cli/operation/OperationManager.java,
> >  line 66
> > <https://reviews.apache.org/r/40500/diff/1/?file=1133036#file1133036line66>
> >
> > any reason we only want sqloperations ?

No special reason. SQLOperations are interesting to show for HS2. Hive commands 
are quick to run. Meta operations, should we show them in HMS web ui?


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40500/#review107283
---


On Nov. 19, 2015, 8:53 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40500/
> ---
> 
> (Updated Nov. 19, 2015, 8:53 p.m.)
> 
> 
> Review request for hive, Szehon Ho and Xuefu Zhang.
> 
> 
> Bugs: HIVE-12338
> https://issues.apache.org/jira/browse/HIVE-12338
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Added web UI to HS2. The UI is similar to those for other Hadoop components.
> The default web UI port is set to 10002, which is configurable. It can be 
> disabled. Currently it shows active sessions and queries. It can also access 
> locals, metrics, and configuration.
> 
> 
> Diffs
> -
> 
>   common/pom.xml cd14581 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2185f85 
>   common/src/java/org/apache/hive/http/AdminAuthorizedServlet.java 
> PRE-CREATION 
>   common/src/java/org/apache/hive/http/ConfServlet.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/HttpServer.java PRE-CREATION 
>   common/src/java/org/apache/hive/http/JMXJsonServlet.java PRE-CREATION 
>   pom.xml c6df4a5 
>   service/pom.xml afa52cf 
>   service/src/java/org/apache/hive/service/cli/operation/Operation.java 
> d13415e 
>   
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
> b0bd351 
>   service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
> 8b42265 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 
> 1ab5652 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 2d784f0 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> d11cf3d 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java b30b6a2 
>   service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp PRE-CREATION 
>   service/src/resources/hive-webapps/hiveserver2/index.html PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/bootstrap-theme.min.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/bootstrap.min.css 
> PRE-CREATION 
>   service/src/resources/hive-webapps/static/css/hive.css PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.eot
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.svg
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.ttf
>  PRE-CREATION 
>   
> service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.woff
>  PRE-CREATION 
>   service/src/resources/hive-webapps/static/hive_logo.jpeg PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40500/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

[jira] [Created] (HIVE-12471) Secure HS2 web UI with SSL and kerberos

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12471:
--

 Summary: Secure HS2 web UI with SSL and kerberos
 Key: HIVE-12471
 URL: https://issues.apache.org/jira/browse/HIVE-12471
 Project: Hive
  Issue Type: Sub-task
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 40500: HIVE-12338 Add webui to HiveServer2

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40500/
---

Review request for hive, Szehon Ho and Xuefu Zhang.


Bugs: HIVE-12338
https://issues.apache.org/jira/browse/HIVE-12338


Repository: hive-git


Description
---

Added web UI to HS2. The UI is similar to those for other Hadoop components.
The default web UI port is set to 10002, which is configurable. It can be 
disabled. Currently it shows active sessions and queries. It can also access 
locals, metrics, and configuration.


Diffs
-

  common/pom.xml cd14581 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2185f85 
  common/src/java/org/apache/hive/http/AdminAuthorizedServlet.java PRE-CREATION 
  common/src/java/org/apache/hive/http/ConfServlet.java PRE-CREATION 
  common/src/java/org/apache/hive/http/HttpServer.java PRE-CREATION 
  common/src/java/org/apache/hive/http/JMXJsonServlet.java PRE-CREATION 
  pom.xml c6df4a5 
  service/pom.xml afa52cf 
  service/src/java/org/apache/hive/service/cli/operation/Operation.java d13415e 
  service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 
b0bd351 
  service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 
8b42265 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionBase.java 
1ab5652 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
2d784f0 
  service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
d11cf3d 
  service/src/java/org/apache/hive/service/server/HiveServer2.java b30b6a2 
  service/src/resources/hive-webapps/hiveserver2/hiveserver2.jsp PRE-CREATION 
  service/src/resources/hive-webapps/hiveserver2/index.html PRE-CREATION 
  service/src/resources/hive-webapps/static/css/bootstrap-theme.min.css 
PRE-CREATION 
  service/src/resources/hive-webapps/static/css/bootstrap.min.css PRE-CREATION 
  service/src/resources/hive-webapps/static/css/hive.css PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.eot
 PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.svg
 PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.ttf
 PRE-CREATION 
  
service/src/resources/hive-webapps/static/fonts/glyphicons-halflings-regular.woff
 PRE-CREATION 
  service/src/resources/hive-webapps/static/hive_logo.jpeg PRE-CREATION 

Diff: https://reviews.apache.org/r/40500/diff/


Testing
---


Thanks,

Jimmy Xiang

Re: file too large - itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz

[Jimmy Xiang]

Done. Branch master-fixed is clean now. What should we do next?

On Fri, Nov 6, 2015 at 9:26 AM, Jimmy Xiang  wrote:

> Let me do that.
>
> On Fri, Nov 6, 2015 at 9:24 AM, Owen O'Malley  wrote:
>
>> Can someone push a fixed master history to "master-fixed"? Then we can
>> start a discussion with infra about replacing master with master-fixed.
>>
>> .. Owen
>>
>> On Fri, Nov 6, 2015 at 9:04 AM, Jimmy Xiang  wrote:
>>
>> > Right. The file is not in source control any more. But it was there at
>> some
>> > moment. That's why it is in the git history. So we need to clean up the
>> git
>> > history.
>> >
>> > To push to a branch other than master, for example, hive-11890, as a
>> > work-around, you can try https://rtyley.github.io/bfg-repo-cleaner/ to
>> do
>> > the clean up first, then push.
>> >
>> > On Fri, Nov 6, 2015 at 9:00 AM, Jimmy Xiang 
>> wrote:
>> >
>> > > I ran into the same problem too. What we need to do is to clean up the
>> > git
>> > > history. However, the master branch can't be force-pushed. I was
>> > wondering
>> > > if this is something the infra team can help.
>> > >
>> > > On Fri, Nov 6, 2015 at 8:54 AM, Xuefu Zhang 
>> wrote:
>> > >
>> > >> I don't quite follow. That file is not in source control, but only
>> > >> downloaded when running test. Why a git push will mass with that
>> file?
>> > >>
>> > >>
>> > >> On Fri, Nov 6, 2015 at 7:44 AM, Owen O'Malley 
>> > wrote:
>> > >>
>> > >> > Github apparently has a limit on pushes to personal accounts that
>> no
>> > >> file
>> > >> > may be larger than 100mb. When I try to push a branch to my
>> personal
>> > >> clone
>> > >> > of hive, I get:
>> > >> >
>> > >> > rockfleet:hive owen$ git push omalley master:hive-11890
>> > >> > > Counting objects: 21159, done.
>> > >> > > Delta compression using up to 4 threads.
>> > >> > > Compressing objects: 100% (6561/6561), done.
>> > >> > > Writing objects: 100% (21159/21159), 110.16 MiB | 3.19 MiB/s,
>> done.
>> > >> > > Total 21159 (delta 11142), reused 18555 (delta 9014)
>> > >> > > remote: error: GH001: Large files detected. You may want to try
>> Git
>> > >> Large
>> > >> > > File Storage - https://git-lfs.github.com.
>> > >> > > remote: error: Trace: 5cfe5408835563c7f661c3086c84930a
>> > >> > > remote: error: See http://git.io/iEPt8g for more information.
>> > >> > > remote: error: File
>> > >> > > itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz is
>> 103.62
>> > >> MB;
>> > >> > > this exceeds GitHub's file size limit of 100.00 MB
>> > >> > > To g...@github.com:omalley/hive.git
>> > >> > >  ! [remote rejected] master -> hive-11890 (pre-receive hook
>> > declined)
>> > >> > > error: failed to push some refs to 'g...@github.com:
>> > omalley/hive.git'
>> > >> >
>> > >> >
>> > >> > This makes it very difficult to work with. Can we do something else
>> > with
>> > >> > the file? If not, can we store it with the git large file storage
>> as
>> > >> > suggested in the error message?
>> > >> >
>> > >> > .. Owen
>> > >> >
>> > >>
>> > >
>> > >
>> >
>>
>
>

Re: file too large - itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz

[Jimmy Xiang]

Let me do that.

On Fri, Nov 6, 2015 at 9:24 AM, Owen O'Malley  wrote:

> Can someone push a fixed master history to "master-fixed"? Then we can
> start a discussion with infra about replacing master with master-fixed.
>
> .. Owen
>
> On Fri, Nov 6, 2015 at 9:04 AM, Jimmy Xiang  wrote:
>
> > Right. The file is not in source control any more. But it was there at
> some
> > moment. That's why it is in the git history. So we need to clean up the
> git
> > history.
> >
> > To push to a branch other than master, for example, hive-11890, as a
> > work-around, you can try https://rtyley.github.io/bfg-repo-cleaner/ to
> do
> > the clean up first, then push.
> >
> > On Fri, Nov 6, 2015 at 9:00 AM, Jimmy Xiang  wrote:
> >
> > > I ran into the same problem too. What we need to do is to clean up the
> > git
> > > history. However, the master branch can't be force-pushed. I was
> > wondering
> > > if this is something the infra team can help.
> > >
> > > On Fri, Nov 6, 2015 at 8:54 AM, Xuefu Zhang 
> wrote:
> > >
> > >> I don't quite follow. That file is not in source control, but only
> > >> downloaded when running test. Why a git push will mass with that file?
> > >>
> > >>
> > >> On Fri, Nov 6, 2015 at 7:44 AM, Owen O'Malley 
> > wrote:
> > >>
> > >> > Github apparently has a limit on pushes to personal accounts that no
> > >> file
> > >> > may be larger than 100mb. When I try to push a branch to my personal
> > >> clone
> > >> > of hive, I get:
> > >> >
> > >> > rockfleet:hive owen$ git push omalley master:hive-11890
> > >> > > Counting objects: 21159, done.
> > >> > > Delta compression using up to 4 threads.
> > >> > > Compressing objects: 100% (6561/6561), done.
> > >> > > Writing objects: 100% (21159/21159), 110.16 MiB | 3.19 MiB/s,
> done.
> > >> > > Total 21159 (delta 11142), reused 18555 (delta 9014)
> > >> > > remote: error: GH001: Large files detected. You may want to try
> Git
> > >> Large
> > >> > > File Storage - https://git-lfs.github.com.
> > >> > > remote: error: Trace: 5cfe5408835563c7f661c3086c84930a
> > >> > > remote: error: See http://git.io/iEPt8g for more information.
> > >> > > remote: error: File
> > >> > > itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz is
> 103.62
> > >> MB;
> > >> > > this exceeds GitHub's file size limit of 100.00 MB
> > >> > > To g...@github.com:omalley/hive.git
> > >> > >  ! [remote rejected] master -> hive-11890 (pre-receive hook
> > declined)
> > >> > > error: failed to push some refs to 'g...@github.com:
> > omalley/hive.git'
> > >> >
> > >> >
> > >> > This makes it very difficult to work with. Can we do something else
> > with
> > >> > the file? If not, can we store it with the git large file storage as
> > >> > suggested in the error message?
> > >> >
> > >> > .. Owen
> > >> >
> > >>
> > >
> > >
> >
>

Re: file too large - itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz

[Jimmy Xiang]

Right. The file is not in source control any more. But it was there at some
moment. That's why it is in the git history. So we need to clean up the git
history.

To push to a branch other than master, for example, hive-11890, as a
work-around, you can try https://rtyley.github.io/bfg-repo-cleaner/ to do
the clean up first, then push.

On Fri, Nov 6, 2015 at 9:00 AM, Jimmy Xiang  wrote:

> I ran into the same problem too. What we need to do is to clean up the git
> history. However, the master branch can't be force-pushed. I was wondering
> if this is something the infra team can help.
>
> On Fri, Nov 6, 2015 at 8:54 AM, Xuefu Zhang  wrote:
>
>> I don't quite follow. That file is not in source control, but only
>> downloaded when running test. Why a git push will mass with that file?
>>
>>
>> On Fri, Nov 6, 2015 at 7:44 AM, Owen O'Malley  wrote:
>>
>> > Github apparently has a limit on pushes to personal accounts that no
>> file
>> > may be larger than 100mb. When I try to push a branch to my personal
>> clone
>> > of hive, I get:
>> >
>> > rockfleet:hive owen$ git push omalley master:hive-11890
>> > > Counting objects: 21159, done.
>> > > Delta compression using up to 4 threads.
>> > > Compressing objects: 100% (6561/6561), done.
>> > > Writing objects: 100% (21159/21159), 110.16 MiB | 3.19 MiB/s, done.
>> > > Total 21159 (delta 11142), reused 18555 (delta 9014)
>> > > remote: error: GH001: Large files detected. You may want to try Git
>> Large
>> > > File Storage - https://git-lfs.github.com.
>> > > remote: error: Trace: 5cfe5408835563c7f661c3086c84930a
>> > > remote: error: See http://git.io/iEPt8g for more information.
>> > > remote: error: File
>> > > itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz is 103.62
>> MB;
>> > > this exceeds GitHub's file size limit of 100.00 MB
>> > > To g...@github.com:omalley/hive.git
>> > >  ! [remote rejected] master -> hive-11890 (pre-receive hook declined)
>> > > error: failed to push some refs to 'g...@github.com:omalley/hive.git'
>> >
>> >
>> > This makes it very difficult to work with. Can we do something else with
>> > the file? If not, can we store it with the git large file storage as
>> > suggested in the error message?
>> >
>> > .. Owen
>> >
>>
>
>

Re: file too large - itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz

[Jimmy Xiang]

I ran into the same problem too. What we need to do is to clean up the git
history. However, the master branch can't be force-pushed. I was wondering
if this is something the infra team can help.

On Fri, Nov 6, 2015 at 8:54 AM, Xuefu Zhang  wrote:

> I don't quite follow. That file is not in source control, but only
> downloaded when running test. Why a git push will mass with that file?
>
>
> On Fri, Nov 6, 2015 at 7:44 AM, Owen O'Malley  wrote:
>
> > Github apparently has a limit on pushes to personal accounts that no file
> > may be larger than 100mb. When I try to push a branch to my personal
> clone
> > of hive, I get:
> >
> > rockfleet:hive owen$ git push omalley master:hive-11890
> > > Counting objects: 21159, done.
> > > Delta compression using up to 4 threads.
> > > Compressing objects: 100% (6561/6561), done.
> > > Writing objects: 100% (21159/21159), 110.16 MiB | 3.19 MiB/s, done.
> > > Total 21159 (delta 11142), reused 18555 (delta 9014)
> > > remote: error: GH001: Large files detected. You may want to try Git
> Large
> > > File Storage - https://git-lfs.github.com.
> > > remote: error: Trace: 5cfe5408835563c7f661c3086c84930a
> > > remote: error: See http://git.io/iEPt8g for more information.
> > > remote: error: File
> > > itests/thirdparty/spark-1.2.0-bin-hadoop2-without-hive.tgz is 103.62
> MB;
> > > this exceeds GitHub's file size limit of 100.00 MB
> > > To g...@github.com:omalley/hive.git
> > >  ! [remote rejected] master -> hive-11890 (pre-receive hook declined)
> > > error: failed to push some refs to 'g...@github.com:omalley/hive.git'
> >
> >
> > This makes it very difficult to work with. Can we do something else with
> > the file? If not, can we store it with the git large file storage as
> > suggested in the error message?
> >
> > .. Owen
> >
>

[jira] [Created] (HIVE-12338) Add webui to HiveServer2

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12338:
--

 Summary: Add webui to HiveServer2
 Key: HIVE-12338
 URL: https://issues.apache.org/jira/browse/HIVE-12338
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


A web ui for HiveServer2 can show some useful information such as:
 
1. Sessions,
2. Queries that are executing on the HS2, their states, starting time, etc.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12318) qtest failing due to NPE in logStats

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12318:
--

 Summary: qtest failing due to NPE in logStats
 Key: HIVE-12318
 URL: https://issues.apache.org/jira/browse/HIVE-12318
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang


{noformat}
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.Operator.logStats(Operator.java:899) ~
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12317) Emit current database in lineage info

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12317:
--

 Summary: Emit current database in lineage info
 Key: HIVE-12317
 URL: https://issues.apache.org/jira/browse/HIVE-12317
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


It will be easier to emit current database info explicitly instead of finding 
out such info from normalized column names.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New Hive Committer - Yongzhi Chen

[Jimmy Xiang]

Congrats!

On Mon, Nov 2, 2015 at 9:43 AM, Xuefu Zhang  wrote:

> Hi all,
>
> Apache Hive PMC has just voted to make Yongzhi Chen a new committer on
> Apache Hive Project.
>
> Please join me in congratulating Yongzhi!
>
> Thanks,
>
> Xuefu
>

Re: Review Request 39741: HIVE-12280 : HiveConnection does not try all possible HS2 in dynamic service discovery mode.

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39741/#review104481
---

Ship it!


Ship It!

- Jimmy Xiang


On Oct. 29, 2015, 6:49 p.m., Szehon Ho wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39741/
> ---
> 
> (Updated Oct. 29, 2015, 6:49 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12280
> https://issues.apache.org/jira/browse/HIVE-12280
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The client calls ZooKeeperHiveClientHelper.configureConnParams() on retry to 
> get a new HS2 host.  But there are some checks to prevent the host/port from 
> getting reset if they are already set, which will make the retries all try on 
> the same host/port.
> 
> 
> Diffs
> -
> 
>   itests/hive-unit/pom.xml 326d646 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestServiceDiscovery.java 
> PRE-CREATION 
>   jdbc/src/java/org/apache/hive/jdbc/ZooKeeperHiveClientHelper.java 6c21423 
> 
> Diff: https://reviews.apache.org/r/39741/diff/
> 
> 
> Testing
> ---
> 
> Added unit test.
> 
> 
> Thanks,
> 
> Szehon Ho
> 
>

[jira] [Created] (HIVE-12287) Lineage for lateral view shows wrong dependencies

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12287:
--

 Summary: Lineage for lateral view shows wrong dependencies
 Key: HIVE-12287
 URL: https://issues.apache.org/jira/browse/HIVE-12287
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


The lineage dependency graph for select from lateral view is wrong.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12278) Skip logging lineage for explain queries

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12278:
--

 Summary: Skip logging lineage for explain queries
 Key: HIVE-12278
 URL: https://issues.apache.org/jira/browse/HIVE-12278
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


For explain queries, we don't generate the lineage info. So we should not try 
to log it at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12268) Context leaks deleteOnExit paths

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12268:
--

 Summary: Context leaks deleteOnExit paths
 Key: HIVE-12268
 URL: https://issues.apache.org/jira/browse/HIVE-12268
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


Long running HS2 saves lots of paths in the FileSystem's deleteOnExit map. We 
should remove those paths already removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12265) Generate lineage info only if requested

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12265:
--

 Summary: Generate lineage info only if requested
 Key: HIVE-12265
 URL: https://issues.apache.org/jira/browse/HIVE-12265
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


If lineage related hook is not configured, we should not generate lineage info.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 39556: HIVE-12187 Release plan once a query is executed

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39556/
---

(Updated Oct. 22, 2015, 4:25 p.m.)


Review request for hive and Chaoyu Tang.


Bugs: HIVE-12187
https://issues.apache.org/jira/browse/HIVE-12187


Repository: hive-git


Description
---

Release plans right after a query is done.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 218b9c8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java e584e6e 

Diff: https://reviews.apache.org/r/39556/diff/


Testing
---

Unit test. Live cluster testing and profiling.


Thanks,

Jimmy Xiang

Review Request 39556: HIVE-12187 Release plan once a query is executed

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39556/
---

Review request for hive and Chaoyu Tang.


Repository: hive-git


Description
---

Release plans right after a query is done.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 218b9c8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java e584e6e 

Diff: https://reviews.apache.org/r/39556/diff/


Testing
---

Unit test. Live cluster testing and profiling.


Thanks,

Jimmy Xiang

Re: [ANNOUNCE] New Hive Committer - Siddharth Seth

[Jimmy Xiang]

Congrats!!

On Wed, Oct 21, 2015 at 2:29 PM, Chaoyu Tang  wrote:

> Congratulations to Siddharth!
>
>
> On Wed, Oct 21, 2015 at 5:25 PM, Xuefu Zhang  wrote:
>
> > Congratulations, Sirddharth!
> >
> > On Wed, Oct 21, 2015 at 2:14 PM, Sergey Shelukhin <
> ser...@hortonworks.com>
> > wrote:
> >
> > > The Apache Hive PMC has voted to make Siddharth Seth a committer on the
> > > Apache Hive Project.
> > >
> > > Please join me in congratulating Sid!
> > >
> > > Thanks,
> > > Sergey.
> > >
> > >
> >
>

Re: [ANNOUNCE] New Hive Committer- Aihua Xu

[Jimmy Xiang]

Congrats!!

On Wed, Oct 21, 2015 at 2:09 PM, Szehon Ho  wrote:

> The Apache Hive PMC has voted to make Aihua Xu a committer on the Apache
> Hive Project.
>
> Please join me in congratulating Aihua!
>
> Thanks,
> Szehon
>

[jira] [Created] (HIVE-12225) LineageCtx should release all resources at clear

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12225:
--

 Summary: LineageCtx should release all resources at clear
 Key: HIVE-12225
 URL: https://issues.apache.org/jira/browse/HIVE-12225
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


Somce maps are not released in clear() method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12200) INSERT INTO table using a select statement w/o a FROM clause fails

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12200:
--

 Summary: INSERT INTO table using a select statement w/o a FROM 
clause fails
 Key: HIVE-12200
 URL: https://issues.apache.org/jira/browse/HIVE-12200
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


Here is the stack trace:

{noformat}
FailedPredicateException(regularBody,{$s.tree.getChild(1) !=null}?)
at 
org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:41047)
at 
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:40222)
at 
org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:40092)
at 
org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1656)
at 
org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1140)
at 
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202)
at 
org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:407)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:312)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1162)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1215)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1091)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1081)
at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:225)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:177)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:388)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:323)
at 
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:731)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:704)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:633)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: ParseException line 1:29 Failed to recognize predicate ''. Failed 
rule: 'regularBody' in statement
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12187) Release plan once a query is executed

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12187:
--

 Summary: Release plan once a query is executed 
 Key: HIVE-12187
 URL: https://issues.apache.org/jira/browse/HIVE-12187
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


Some clients leave query operations open for a while so that they can retrieve 
the query results later. That means the allocated memory will be kept around 
too. We should release those resources not needed for query execution any more 
once it is executed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12046) Re-create spark client if connection is dropped

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-12046:
--

 Summary: Re-create spark client if connection is dropped
 Key: HIVE-12046
 URL: https://issues.apache.org/jira/browse/HIVE-12046
 Project: Hive
  Issue Type: Bug
  Components: Spark
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


Currently, if the connection to the spark cluster is dropped, the spark client 
will stay in a bad state. A new Hive session is needed to re-establish the 
connection. It is better to auto reconnect in this case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11984) Add HS2 open operation metrics

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11984:
--

 Summary: Add HS2 open operation metrics
 Key: HIVE-11984
 URL: https://issues.apache.org/jira/browse/HIVE-11984
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


Some metrics for open operations should be helpful to track operations not 
closed/cancelled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11946) TestNotificationListener is flaky

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11946:
--

 Summary: TestNotificationListener is flaky
 Key: HIVE-11946
 URL: https://issues.apache.org/jira/browse/HIVE-11946
 Project: Hive
  Issue Type: Test
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


{noformat}
expected:<[CREATE_DATABASE, CREATE_TABLE, ADD_PARTITION, ALTER_PARTITION, 
DROP_PARTITION, ALTER_TABLE, DROP_TABLE, DROP_DATABASE]> but 
was:<[CREATE_DATABASE, CREATE_TABLE, ADD_PARTITION, ALTER_PARTITION, 
DROP_PARTITION, ALTER_TABLE, DROP_TABLE]>

Stacktrace

java.lang.AssertionError: expected:<[CREATE_DATABASE, CREATE_TABLE, 
ADD_PARTITION, ALTER_PARTITION, DROP_PARTITION, ALTER_TABLE, DROP_TABLE, 
DROP_DATABASE]> but was:<[CREATE_DATABASE, CREATE_TABLE, ADD_PARTITION, 
ALTER_PARTITION, DROP_PARTITION, ALTER_TABLE, DROP_TABLE]>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hive.hcatalog.listener.TestNotificationListener.tearDown(TestNotificationListener.java:114)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11939) TxnDbUtil should turn off jdbc auto commit

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11939:
--

 Summary: TxnDbUtil should turn off jdbc auto commit
 Key: HIVE-11939
 URL: https://issues.apache.org/jira/browse/HIVE-11939
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


TxnDbUtil uses jdbc transactions, but doesn't turn off auto commit. So some 
TestStreaming tests are flaky. For example,
{noformat}
testTransactionBatchAbortAndCommit(org.apache.hive.hcatalog.streaming.TestStreaming)
  Time elapsed: 0.011 sec  <<< ERROR!
java.sql.SQLException: Table/View 'TXNS' already exists in Schema 'APP'.
at org.apache.derby.iapi.error.StandardException.newException(Unknown 
Source)
at 
org.apache.derby.impl.sql.catalog.DataDictionaryImpl.duplicateDescriptorException(Unknown
 Source)
at 
org.apache.derby.impl.sql.catalog.DataDictionaryImpl.addDescriptor(Unknown 
Source)
at 
org.apache.derby.impl.sql.execute.CreateTableConstantAction.executeConstantAction(Unknown
 Source)
at org.apache.derby.impl.sql.execute.MiscResultSet.open(Unknown Source)
at 
org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown 
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown 
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
at 
org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:72)
at 
org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:131)
at 
org.apache.hive.hcatalog.streaming.TestStreaming.(TestStreaming.java:160)

{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 38442: HIVE-11834 Lineage doesn't work with dynamic partitioning query

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38442/
---

Review request for hive and Szehon Ho.


Bugs: HIVE-11834
https://issues.apache.org/jira/browse/HIVE-11834


Repository: hive-git


Description
---

Fixed a bug in handling dynamic partitioning query output fields.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageLogger.java f615d81 
  ql/src/test/queries/clientpositive/lineage3.q c24ff7d 
  ql/src/test/results/clientpositive/lineage3.q.out 708abee 

Diff: https://reviews.apache.org/r/38442/diff/


Testing
---

Unit tests.


Thanks,

Jimmy Xiang

[jira] [Created] (HIVE-11834) Lineage doesn't work with dynamic partitioning query

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11834:
--

 Summary: Lineage doesn't work with dynamic partitioning query
 Key: HIVE-11834
 URL: https://issues.apache.org/jira/browse/HIVE-11834
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


As Mark found out,
https://issues.apache.org/jira/browse/HIVE-11139?focusedCommentId=14745937&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14745937

This is indeed a code bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11817) Window function max NullPointerException

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11817:
--

 Summary: Window function max NullPointerException
 Key: HIVE-11817
 URL: https://issues.apache.org/jira/browse/HIVE-11817
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


This query
{noformat}
select key, max(value) over (order by key rows between 10 preceding and 20 
following) from src1 where length(key) > 10;
{noformat}
fails with NPE:
{noformat}
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMax$MaxStreamingFixedWindow.terminate(GenericUDAFMax.java:290)
 
at 
org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.finishPartition(WindowingTableFunction.java:477)
 
at 
org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:337)
 
at 
org.apache.hadoop.hive.ql.exec.PTFOperator.closeOp(PTFOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:617)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:631)
at 
org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:278)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11814) Emit query time in lineage info

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11814:
--

 Summary: Emit query time in lineage info
 Key: HIVE-11814
 URL: https://issues.apache.org/jira/browse/HIVE-11814
 Project: Hive
  Issue Type: Improvement
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


Currently, we emit query start time, not the query duration. It is nice to have 
it too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11771) Parquet timestamp conversion errors

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11771:
--

 Summary: Parquet timestamp conversion errors
 Key: HIVE-11771
 URL: https://issues.apache.org/jira/browse/HIVE-11771
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


We have some problem to read timestamp written to parquet file by other tools. 
The value is wrong after the conversion (not the same as it is meant to be).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11737) IndexOutOfBounds compiling query with duplicated groupby keys

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11737:
--

 Summary: IndexOutOfBounds compiling query with duplicated groupby 
keys
 Key: HIVE-11737
 URL: https://issues.apache.org/jira/browse/HIVE-11737
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


{noformat}
SELECT
tinyint_col_7,
MIN(timestamp_col_1) AS timestamp_col,
MAX(LEAST(CAST(COALESCE(int_col_5, -279) AS int), CAST(COALESCE(tinyint_col_7, 
476) AS int))) AS int_col,
tinyint_col_7 AS int_col_1,
LEAST(CAST(COALESCE(int_col_5, -279) AS int), CAST(COALESCE(tinyint_col_7, 476) 
AS int)) AS int_col_2
FROM table_3
GROUP BY
tinyint_col_7,
tinyint_col_7,
LEAST(CAST(COALESCE(int_col_5, -279) AS int), CAST(COALESCE(tinyint_col_7, 476) 
AS int))
{noformat}

Query compilation fails:
{noformat}
Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
at java.util.ArrayList.rangeCheck(ArrayList.java:635)
at java.util.ArrayList.get(ArrayList.java:411)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanReduceSinkOperator(SemanticAnalyzer.java:4633)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:5630)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8987)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9864)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9757)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10193)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10204)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10121)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:421)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1110)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1104)
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11712) Duplicate groupby keys cause ClassCastException

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11712:
--

 Summary: Duplicate groupby keys cause ClassCastException
 Key: HIVE-11712
 URL: https://issues.apache.org/jira/browse/HIVE-11712
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


With duplicate groupby keys, we could use wrong object inspectors for some 
groupby expressions, and lead to ClassCastException, for example, 

{noformat}
explain
SELECT distinct s1.customer_name as x, s1.customer_name as y
FROM default.testv1_staples s1 join default.src s2 on s1.customer_name = s2.key
HAVING (
(SUM(s1.customer_balance) <= 4074689.00041)
AND (AVG(s1.discount) <= 822)
AND (COUNT(s2.value) > 4)
{noformat}

will lead to

{noformat}
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableShortObjectInspector
 cannot be cast to 
org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector
at 
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFAverage$AbstractGenericUDAFAverageEvaluator.init(GenericUDAFAverage.java:374)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFInfo(SemanticAnalyzer.java:3887)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanGroupByOperator1(SemanticAnalyzer.java:4354)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:5644)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8977)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9849)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9742)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10178)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10189)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10106)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11620) Fix several qtest output order

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11620:
--

 Summary: Fix several qtest output order
 Key: HIVE-11620
 URL: https://issues.apache.org/jira/browse/HIVE-11620
 Project: Hive
  Issue Type: Test
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


selectDistinctStar.q
unionall_unbalancedppd.q
vector_cast_constant.q



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 37591: HIVE-11586 ObjectInspectorFactory.getReflectionObjectInspector is not thread-safe

[Jimmy Xiang]

> On Aug. 19, 2015, 6:17 p.m., Xuefu Zhang wrote:
> > Thanks, guys. However, I do see that the solution seems complicated and 
> > error-prone. It will be great if we can simplify this.
> 
> Jimmy Xiang wrote:
> To support recursive type, so far, I don't see a better solution. We can 
> enhance it if someone comes up a better solution later on. Thanks.
> 
> Xuefu Zhang wrote:
> I think the part of complexity may come from the fact that the 
> synchronization happens in different files. It might become cleaner if we can 
> somehow refector the code so that synchonization blocks stay in the same file.
> 
> Jimmy Xiang wrote:
> That's an interesting idea. But ThriftUnionObjectInspector and 
> ReflectionStructObjectInspector have different init logic. How can we 
> refactor them into a same file? That may introduce more changes, right?
> 
> Xuefu Zhang wrote:
> I don't know. I was thinking that some helper methods in a util class 
> should help.

But it will make things more complex.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37591/#review95866
-------


On Aug. 18, 2015, 9:51 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37591/
> ---
> 
> (Updated Aug. 18, 2015, 9:51 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11586
> https://issues.apache.org/jira/browse/HIVE-11586
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Made ObjectInspectorFactory.getReflectionObjectInspector thread-safe. When it 
> returns, the object inspector is fully initialized.
> 
> 
> Diffs
> -
> 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
>  97bb715 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
>  78e6066 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java
>  600abbb 
>   
> serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
>  e2408c6 
> 
> Diff: https://reviews.apache.org/r/37591/diff/
> 
> 
> Testing
> ---
> 
> Unit test
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 37591: HIVE-11586 ObjectInspectorFactory.getReflectionObjectInspector is not thread-safe

[Jimmy Xiang]

> On Aug. 19, 2015, 6:17 p.m., Xuefu Zhang wrote:
> > Thanks, guys. However, I do see that the solution seems complicated and 
> > error-prone. It will be great if we can simplify this.
> 
> Jimmy Xiang wrote:
> To support recursive type, so far, I don't see a better solution. We can 
> enhance it if someone comes up a better solution later on. Thanks.
> 
> Xuefu Zhang wrote:
> I think the part of complexity may come from the fact that the 
> synchronization happens in different files. It might become cleaner if we can 
> somehow refector the code so that synchonization blocks stay in the same file.

That's an interesting idea. But ThriftUnionObjectInspector and 
ReflectionStructObjectInspector have different init logic. How can we refactor 
them into a same file? That may introduce more changes, right?


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37591/#review95866
-------


On Aug. 18, 2015, 9:51 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37591/
> ---
> 
> (Updated Aug. 18, 2015, 9:51 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11586
> https://issues.apache.org/jira/browse/HIVE-11586
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Made ObjectInspectorFactory.getReflectionObjectInspector thread-safe. When it 
> returns, the object inspector is fully initialized.
> 
> 
> Diffs
> -
> 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
>  97bb715 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
>  78e6066 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java
>  600abbb 
>   
> serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
>  e2408c6 
> 
> Diff: https://reviews.apache.org/r/37591/diff/
> 
> 
> Testing
> ---
> 
> Unit test
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 37591: HIVE-11586 ObjectInspectorFactory.getReflectionObjectInspector is not thread-safe

[Jimmy Xiang]

> On Aug. 19, 2015, 6:17 p.m., Xuefu Zhang wrote:
> > Thanks, guys. However, I do see that the solution seems complicated and 
> > error-prone. It will be great if we can simplify this.

To support recursive type, so far, I don't see a better solution. We can 
enhance it if someone comes up a better solution later on. Thanks.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37591/#review95866
---


On Aug. 18, 2015, 9:51 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37591/
> ---
> 
> (Updated Aug. 18, 2015, 9:51 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11586
> https://issues.apache.org/jira/browse/HIVE-11586
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Made ObjectInspectorFactory.getReflectionObjectInspector thread-safe. When it 
> returns, the object inspector is fully initialized.
> 
> 
> Diffs
> -
> 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
>  97bb715 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
>  78e6066 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java
>  600abbb 
>   
> serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
>  e2408c6 
> 
> Diff: https://reviews.apache.org/r/37591/diff/
> 
> 
> Testing
> ---
> 
> Unit test
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 37591: HIVE-11586 ObjectInspectorFactory.getReflectionObjectInspector is not thread-safe

[Jimmy Xiang]

> On Aug. 18, 2015, 10:51 p.m., Szehon Ho wrote:
> > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java,
> >  line 90
> > <https://reviews.apache.org/r/37591/diff/2/?file=1043884#file1043884line90>
> >
> > Seems a little risky to me, what if there are some bad GC pause?  
> > 
> > If it's fully-inited it should get notified , so how about a longer 
> > timeout?
> 
> Szehon Ho wrote:
> Sorry, my question should be, why can't we sleep the thread, and wait for 
> it to get notified?
> 
> Jimmy Xiang wrote:
> We have a while loop, so it is fine to take longer than this much time to 
> init. Here, wait() achieves the same effect as sleep and get notified. It's 
> kind of sleep for up to that much time unless it is got notified earlier.
> 
> Szehon Ho wrote:
> I mean, should we wait without a timeout then?  Not sure if we will ever 
> miss a notify?
> 
> Jimmy Xiang wrote:
> I see. That should work too. With a timeout, so that we can periodically 
> run a check and throw RuntimeException if things go wrong.
> 
> Szehon Ho wrote:
> I'm not so sure, I think it just does the same check in a while loop 
> without any other exit condition, right?  Interrupt exception will happen 
> anyway regardless of the timeout.

Right, if the init is ok, both work the same way. When the init fails, using 
wait() is a little different, it could think the init is ok.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37591/#review95789
---


On Aug. 18, 2015, 9:51 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37591/
> ---
> 
> (Updated Aug. 18, 2015, 9:51 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11586
> https://issues.apache.org/jira/browse/HIVE-11586
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Made ObjectInspectorFactory.getReflectionObjectInspector thread-safe. When it 
> returns, the object inspector is fully initialized.
> 
> 
> Diffs
> -
> 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
>  97bb715 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
>  78e6066 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java
>  600abbb 
>   
> serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
>  e2408c6 
> 
> Diff: https://reviews.apache.org/r/37591/diff/
> 
> 
> Testing
> ---
> 
> Unit test
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 37591: HIVE-11586 ObjectInspectorFactory.getReflectionObjectInspector is not thread-safe

[Jimmy Xiang]

> On Aug. 18, 2015, 10:51 p.m., Szehon Ho wrote:
> > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java,
> >  line 90
> > <https://reviews.apache.org/r/37591/diff/2/?file=1043884#file1043884line90>
> >
> > Seems a little risky to me, what if there are some bad GC pause?  
> > 
> > If it's fully-inited it should get notified , so how about a longer 
> > timeout?
> 
> Szehon Ho wrote:
> Sorry, my question should be, why can't we sleep the thread, and wait for 
> it to get notified?
> 
> Jimmy Xiang wrote:
> We have a while loop, so it is fine to take longer than this much time to 
> init. Here, wait() achieves the same effect as sleep and get notified. It's 
> kind of sleep for up to that much time unless it is got notified earlier.
> 
> Szehon Ho wrote:
> I mean, should we wait without a timeout then?  Not sure if we will ever 
> miss a notify?

I see. That should work too. With a timeout, so that we can periodically run a 
check and throw RuntimeException if things go wrong.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37591/#review95789
---


On Aug. 18, 2015, 9:51 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37591/
> ---
> 
> (Updated Aug. 18, 2015, 9:51 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11586
> https://issues.apache.org/jira/browse/HIVE-11586
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Made ObjectInspectorFactory.getReflectionObjectInspector thread-safe. When it 
> returns, the object inspector is fully initialized.
> 
> 
> Diffs
> -
> 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
>  97bb715 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
>  78e6066 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java
>  600abbb 
>   
> serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
>  e2408c6 
> 
> Diff: https://reviews.apache.org/r/37591/diff/
> 
> 
> Testing
> ---
> 
> Unit test
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 37591: HIVE-11586 ObjectInspectorFactory.getReflectionObjectInspector is not thread-safe

[Jimmy Xiang]

> On Aug. 18, 2015, 10:51 p.m., Szehon Ho wrote:
> > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java,
> >  line 90
> > <https://reviews.apache.org/r/37591/diff/2/?file=1043884#file1043884line90>
> >
> > Seems a little risky to me, what if there are some bad GC pause?  
> > 
> > If it's fully-inited it should get notified , so how about a longer 
> > timeout?
> 
> Szehon Ho wrote:
> Sorry, my question should be, why can't we sleep the thread, and wait for 
> it to get notified?

We have a while loop, so it is fine to take longer than this much time to init. 
Here, wait() achieves the same effect as sleep and get notified. It's kind of 
sleep for up to that much time unless it is got notified earlier.


> On Aug. 18, 2015, 10:51 p.m., Szehon Ho wrote:
> > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java,
> >  line 165
> > <https://reviews.apache.org/r/37591/diff/2/?file=1043885#file1043885line165>
> >
> > Very basic question I'm sure you thought of, why can't we synchronize 
> > the entire init() method?  I assume this method will init all the children. 
> >  Then we won't need the isFullyInit methods?

Since some type could refer to itself recursively, for example the MyStruct 
class in the test, if we synchronize the entire init() method without the 
isFullyInited method, we will run into deadlocks.


- Jimmy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37591/#review95789
---


On Aug. 18, 2015, 9:51 p.m., Jimmy Xiang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/37591/
> ---
> 
> (Updated Aug. 18, 2015, 9:51 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-11586
> https://issues.apache.org/jira/browse/HIVE-11586
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Made ObjectInspectorFactory.getReflectionObjectInspector thread-safe. When it 
> returns, the object inspector is fully initialized.
> 
> 
> Diffs
> -
> 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
>  97bb715 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
>  78e6066 
>   
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java
>  600abbb 
>   
> serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
>  e2408c6 
> 
> Diff: https://reviews.apache.org/r/37591/diff/
> 
> 
> Testing
> ---
> 
> Unit test
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>

Re: Review Request 37591: HIVE-11586 ObjectInspectorFactory.getReflectionObjectInspector is not thread-safe

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37591/
---

(Updated Aug. 18, 2015, 9:51 p.m.)


Review request for hive.


Bugs: HIVE-11586
https://issues.apache.org/jira/browse/HIVE-11586


Repository: hive-git


Description
---

Made ObjectInspectorFactory.getReflectionObjectInspector thread-safe. When it 
returns, the object inspector is fully initialized.


Diffs (updated)
-

  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
 97bb715 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
 78e6066 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java
 600abbb 
  
serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
 e2408c6 

Diff: https://reviews.apache.org/r/37591/diff/


Testing
---

Unit test


Thanks,

Jimmy Xiang

Review Request 37591: HIVE-11586 ObjectInspectorFactory.getReflectionObjectInspector is not thread-safe

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37591/
---

Review request for hive.


Bugs: HIVE-11586
https://issues.apache.org/jira/browse/HIVE-11586


Repository: hive-git


Description
---

Made ObjectInspectorFactory.getReflectionObjectInspector thread-safe. When it 
returns, the object inspector is fully initialized.


Diffs
-

  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
 97bb715 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
 78e6066 
  
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java
 600abbb 
  
serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestReflectionObjectInspectors.java
 e2408c6 

Diff: https://reviews.apache.org/r/37591/diff/


Testing
---

Unit test


Thanks,

Jimmy Xiang

[jira] [Created] (HIVE-11586) ObjectInspectorFactory.getReflectionObjectInspector is not thread-safe

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11586:
--

 Summary: ObjectInspectorFactory.getReflectionObjectInspector is 
not thread-safe
 Key: HIVE-11586
 URL: https://issues.apache.org/jira/browse/HIVE-11586
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang


ObjectInspectorFactory#getReflectionObjectInspectorNoCache addes newly create 
object inspector to the cache before calling its init() method, to allow 
reusing the cache when dealing with recursive types. So a second thread can 
then call getReflectionObjectInspector and fetch an uninitialized instance of 
ReflectionStructObjectInspector.

Another issue is that if two threads calls 
ObjectInspectorFactory.getReflectionObjectInspector at the same time. One 
thread could get an object inspector not in the cache, i.e. they could both 
call getReflectionObjectInspectorNoCache() but only one will put the new object 
inspector to cache successfully.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-11580) ThriftUnionObjectInspector#toString throws NPE

[Jimmy Xiang (JIRA)]

Jimmy Xiang created HIVE-11580:
--

 Summary: ThriftUnionObjectInspector#toString throws NPE
 Key: HIVE-11580
 URL: https://issues.apache.org/jira/browse/HIVE-11580
 Project: Hive
  Issue Type: Bug
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor


ThriftUnionObjectInspector uses toString from StructObjectInspector, which 
accesses uninitialized member variable fields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 37375: HIVE-11464 lineage info missing if there are multiple outputs

[Jimmy Xiang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/37375/
---

Review request for hive and Szehon Ho.


Bugs: HIVE-11464
https://issues.apache.org/jira/browse/HIVE-11464


Repository: hive-git


Description
---

Fixed missed lineage info when there are multiple inserts in one query 
statement. In generating the depedency map, we record the insert target tables 
and the corresponding select operators. We used to assume there is just one 
final select operator. It could be several, for example, in the multiple 
inserts case.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java e7b7b55 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageInfo.java fe0841e 
  ql/src/java/org/apache/hadoop/hive/ql/hooks/LineageLogger.java d615372 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/ExprProcFactory.java 
455a525 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/LineageCtx.java 
d26d8da 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/lineage/OpProcFactory.java 
f670db8 
  ql/src/test/queries/clientpositive/lineage3.q 53fff0f 
  ql/src/test/results/clientpositive/lineage3.q.out 75d88f8 

Diff: https://reviews.apache.org/r/37375/diff/


Testing
---

Unit tests


Thanks,

Jimmy Xiang