Review Request 52978: HIVE-14459 TestBeeLineDriver - migration and re-enable

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52978/
---

Review request for hive, Zoltan Haindrich, Marta Kuczora, Miklos Csanady, 
Prasanth_J, Sergey Shelukhin, Sergio Pena, Siddharth Seth, and Barna Zsombor 
Klara.


Bugs: HIVE-14459
https://issues.apache.org/jira/browse/HIVE-14459


Repository: hive-git


Description
---

Kept the changes minimal with the sole goal to be able to run the BeeLine query 
tests multiple times successfully.
- Enabled the driver
- Modified the regexps to hide when comparing the results
- Configured to run only 1 qtest file - so we can test, and could decide later 
of the beeline testing scope
- Added required dependencies to pom
- Added specific results dir for beeline q.out-s


Diffs
-

  beeline/src/java/org/apache/hive/beeline/util/QFileClient.java 81f1b0e 
  
itests/qtest/src/test/java/org/apache/hadoop/hive/cli/DisabledTestBeeLineDriver.java
 cb276e6 
  itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestBeeLineDriver.java 
PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 97e310d 
  itests/util/pom.xml 6d93dc1 
  itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
366c7b4 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
 e5144e3 
  ql/src/test/results/clientpositive/beeline/escape_comments.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/52978/diff/


Testing
---

Manually on my computer several runs.
Waiting for the QA tests


Thanks,

Peter Vary

Re: Review Request 52978: HIVE-14459 TestBeeLineDriver - migration and re-enable

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52978/#review153046
---


Ship it!




Hello Peter!

Beeline tests are coming back into business? That's great! ;)
I think its a good idea to add just one case for now - that way we can ensure 
that its in working order; its a big step forward from not running any of them.

cheers,
Zoltan

- Zoltan Haindrich


On Oct. 18, 2016, 7:02 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52978/
> ---
> 
> (Updated Oct. 18, 2016, 7:02 a.m.)
> 
> 
> Review request for hive, Zoltan Haindrich, Marta Kuczora, Miklos Csanady, 
> Prasanth_J, Sergey Shelukhin, Sergio Pena, Siddharth Seth, and Barna Zsombor 
> Klara.
> 
> 
> Bugs: HIVE-14459
> https://issues.apache.org/jira/browse/HIVE-14459
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Kept the changes minimal with the sole goal to be able to run the BeeLine 
> query tests multiple times successfully.
> - Enabled the driver
> - Modified the regexps to hide when comparing the results
> - Configured to run only 1 qtest file - so we can test, and could decide 
> later of the beeline testing scope
> - Added required dependencies to pom
> - Added specific results dir for beeline q.out-s
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/util/QFileClient.java 81f1b0e 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/DisabledTestBeeLineDriver.java
>  cb276e6 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestBeeLineDriver.java 
> PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 97e310d 
>   itests/util/pom.xml 6d93dc1 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 366c7b4 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
>  e5144e3 
>   ql/src/test/results/clientpositive/beeline/escape_comments.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52978/diff/
> 
> 
> Testing
> ---
> 
> Manually on my computer several runs.
> Waiting for the QA tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

RE: Start HiveServer2 with Kerberos meet FATAL error

[Chen, Junjie J]

Please try to use full qualified name of host,  such as 
hive2/tempt48.hadoop@hadoop.com

-Original Message-
From: Micro dong [mailto:microle.d...@gmail.com] 
Sent: Tuesday, October 18, 2016 2:37 PM
To: dev@hive.apache.org
Subject: Re: Start HiveServer2 with Kerberos meet FATAL error

I follow your suggestion and it also meet the same error too.

*start hiveservice*
/home/work/software/hive/bin/hive --service hiveserver2


*klist output*
Ticket cache: FILE:/tmp/krb5cc_31100
Default principal: hive2/temp...@hadoop.com

Valid starting ExpiresService principal
10/18/16 14:23:04  10/19/16 14:23:04  krbtgt/hadoop@hadoop.com renew until 
10/28/16 14:23:04

2016-10-18 12:07 GMT+08:00 Vivek Shrivastava :

> can you please change this line
>
> hive2/_h...@hadoop.com * with
> hive2/_h...@hadoop.com*
>
> How are you starting the service, What is the klist output?
>
>
> On Mon, Oct 17, 2016 at 11:09 PM, Micro dong 
> wrote:
>
> > I'm trying to configure HiveServer2(hive-1.2.1)  With Kerberos。Here 
> > is my Hive's configuration file。
> >  **
> > *hive.server2.authentication*
> > *KERBEROS*
> > *  *
> > *  *
> > *hive.server2.authentication.kerberos.principal*
> > *hive2/_h...@hadoop.com *
> > *  *
> > *  *
> > *hive.server2.authentication.kerberos.keytab*
> > */home/work/software/hive/conf/hive.keytab*
> > *  *
> >
> > the keytab file is in its location, its owner is work. But when I 
> > try to start the HiveServer2, I see this message on the log:
> >
> > *2016-10-18 10:20:24,867 FATAL [Thread-9]: thrift.ThriftCLIService
> > (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2:
> could
> > not start ThriftBinaryCLIService*
> > *javax.security.auth.login.LoginException: Kerberos principal should
> have
> > 3
> > parts: work**  at *
> > *org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(
> > HiveAuthFactory.java:147)*
> > *atorg.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(
> > ThriftBinaryCLIService.java:58)*
> > *at java.lang.Thread.run(Thread.java:722)*
> >
> > Here *work* is my unix login name. Any help would be highly appreciated.
> > Best regards
> >
>



--
英者自知，雄者自胜。

Re: Review Request 52978: HIVE-14459 TestBeeLineDriver - migration and re-enable

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52978/#review153051
---


Ship it!




Thanks for the patch. +1

- Barna Zsombor Klara


On Oct. 18, 2016, 7:02 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52978/
> ---
> 
> (Updated Oct. 18, 2016, 7:02 a.m.)
> 
> 
> Review request for hive, Zoltan Haindrich, Marta Kuczora, Miklos Csanady, 
> Prasanth_J, Sergey Shelukhin, Sergio Pena, Siddharth Seth, and Barna Zsombor 
> Klara.
> 
> 
> Bugs: HIVE-14459
> https://issues.apache.org/jira/browse/HIVE-14459
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Kept the changes minimal with the sole goal to be able to run the BeeLine 
> query tests multiple times successfully.
> - Enabled the driver
> - Modified the regexps to hide when comparing the results
> - Configured to run only 1 qtest file - so we can test, and could decide 
> later of the beeline testing scope
> - Added required dependencies to pom
> - Added specific results dir for beeline q.out-s
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/util/QFileClient.java 81f1b0e 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/DisabledTestBeeLineDriver.java
>  cb276e6 
>   
> itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestBeeLineDriver.java 
> PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 97e310d 
>   itests/util/pom.xml 6d93dc1 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
> 366c7b4 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
>  e5144e3 
>   ql/src/test/results/clientpositive/beeline/escape_comments.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52978/diff/
> 
> 
> Testing
> ---
> 
> Manually on my computer several runs.
> Waiting for the QA tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Review Request 52981: HIVE-14679: csv2/tsv2 output format disables quoting by default and it's difficult to enable

[Jianguo Tian]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52981/
---

Review request for hive and cheng xu.


Summary (updated)
-

HIVE-14679: csv2/tsv2 output format disables quoting by default and it's 
difficult to enable


Bugs: HIVE-14679
https://issues.apache.org/jira/browse/HIVE-14679


Repository: hive-git


Description
---

1. Quoting should be enabled by default for csv2, tsv2 and dsv.
2. Disabling quoting should be possible using a beeline argument.


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java 79922d2 
  beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 57b9c46 
  beeline/src/java/org/apache/hive/beeline/Commands.java 039e354 
  beeline/src/java/org/apache/hive/beeline/SeparatedValuesOutputFormat.java 
66d9fd0 
  beeline/src/main/resources/BeeLine.properties ad79c01 

Diff: https://reviews.apache.org/r/52981/diff/


Testing
---


Thanks,

Jianguo Tian

Re: Start HiveServer2 with Kerberos meet FATAL error

[Micro dong]

tempt48  is the local machine's hostname
which is configured in /etc/hosts.

2016-10-18 15:33 GMT+08:00 Chen, Junjie J :

> Please try to use full qualified name of host,  such as hive2/
> tempt48.hadoop@hadoop.com
>
> -Original Message-
> From: Micro dong [mailto:microle.d...@gmail.com]
> Sent: Tuesday, October 18, 2016 2:37 PM
> To: dev@hive.apache.org
> Subject: Re: Start HiveServer2 with Kerberos meet FATAL error
>
> I follow your suggestion and it also meet the same error too.
>
> *start hiveservice*
> /home/work/software/hive/bin/hive --service hiveserver2
>
>
> *klist output*
> Ticket cache: FILE:/tmp/krb5cc_31100
> Default principal: hive2/temp...@hadoop.com
>
> Valid starting ExpiresService principal
> 10/18/16 14:23:04  10/19/16 14:23:04  krbtgt/hadoop@hadoop.com renew
> until 10/28/16 14:23:04
>
> 2016-10-18 12:07 GMT+08:00 Vivek Shrivastava :
>
> > can you please change this line
> >
> > hive2/_h...@hadoop.com * with
> > hive2/_h...@hadoop.com*
> >
> > How are you starting the service, What is the klist output?
> >
> >
> > On Mon, Oct 17, 2016 at 11:09 PM, Micro dong 
> > wrote:
> >
> > > I'm trying to configure HiveServer2(hive-1.2.1)  With Kerberos。Here
> > > is my Hive's configuration file。
> > >  **
> > > *hive.server2.authentication*
> > > *KERBEROS*
> > > *  *
> > > *  *
> > > *hive.server2.authentication.kerberos.principal*
> > > *hive2/_h...@hadoop.com *
> > > *  *
> > > *  *
> > > *hive.server2.authentication.kerberos.keytab*
> > > */home/work/software/hive/conf/hive.keytab*
> > > *  *
> > >
> > > the keytab file is in its location, its owner is work. But when I
> > > try to start the HiveServer2, I see this message on the log:
> > >
> > > *2016-10-18 10:20:24,867 FATAL [Thread-9]: thrift.ThriftCLIService
> > > (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2:
> > could
> > > not start ThriftBinaryCLIService*
> > > *javax.security.auth.login.LoginException: Kerberos principal should
> > have
> > > 3
> > > parts: work**  at *
> > > *org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(
> > > HiveAuthFactory.java:147)*
> > > *atorg.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(
> > > ThriftBinaryCLIService.java:58)*
> > > *at java.lang.Thread.run(Thread.java:722)*
> > >
> > > Here *work* is my unix login name. Any help would be highly
> appreciated.
> > > Best regards
> > >
> >
>
>
>
> --
> 英者自知，雄者自胜。
>



-- 
英者自知，雄者自胜。

Re: Review Request 52923: HIVE-14979 Removing stale Zookeeper locks at HiveServer2 initialization

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52923/
---

(Updated Oct. 18, 2016, 8:49 a.m.)


Review request for hive, Ashutosh Chauhan, Marta Kuczora, Miklos Csanady, namit 
jain, Sergio Pena, and Barna Zsombor Klara.


Bugs: HIVE-14979
https://issues.apache.org/jira/browse/HIVE-14979


Repository: hive-git


Description
---

Adding a new configuration option to HiveConf to signal whether stale lock 
removal is requested on startup.
Adding a new method to ZooKeeperHiveLockManager to remove stale locks
Modifying the HiveServer2 to instantiate a lock manager and call the new method 
if defined by the configuration.

Please take extra care when reviewing these:
- Modifying the lock fetching method to use the clientIp from the lock, and not 
update with the current ip - Not sure why it was done before
- Instantiation of the lock manager - I might not chose the best method for it

Open for any suggestions :)

Thanks,
Peter


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8ffae3b 
  
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
 14d0ef4 
  
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
 3f9926e 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 590b1f3 

Diff: https://reviews.apache.org/r/52923/diff/


Testing
---

Created 2 unit test cases:
- Removing own locks
- Not removing other server's locks

Manually tested the Lock manager instantiation method on HiveServer2


Thanks,

Peter Vary

Re: Review Request 52923: HIVE-14979 Removing stale Zookeeper locks at HiveServer2 initialization

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52923/
---

(Updated Oct. 18, 2016, 9:23 a.m.)


Review request for hive, Ashutosh Chauhan, Marta Kuczora, Miklos Csanady, namit 
jain, Sergio Pena, and Barna Zsombor Klara.


Changes
---

Created a hard limit on the test infrastructure creation - if not succeded then 
giving up


Bugs: HIVE-14979
https://issues.apache.org/jira/browse/HIVE-14979


Repository: hive-git


Description
---

Adding a new configuration option to HiveConf to signal whether stale lock 
removal is requested on startup.
Adding a new method to ZooKeeperHiveLockManager to remove stale locks
Modifying the HiveServer2 to instantiate a lock manager and call the new method 
if defined by the configuration.

Please take extra care when reviewing these:
- Modifying the lock fetching method to use the clientIp from the lock, and not 
update with the current ip - Not sure why it was done before
- Instantiation of the lock manager - I might not chose the best method for it

Open for any suggestions :)

Thanks,
Peter


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8ffae3b 
  
ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
 14d0ef4 
  
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
 3f9926e 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 590b1f3 

Diff: https://reviews.apache.org/r/52923/diff/


Testing
---

Created 2 unit test cases:
- Removing own locks
- Not removing other server's locks

Manually tested the Lock manager instantiation method on HiveServer2


Thanks,

Peter Vary

Re: Review Request 52923: HIVE-14979 Removing stale Zookeeper locks at HiveServer2 initialization

[Peter Vary]

> On Oct. 17, 2016, 12:47 p.m., Barna Zsombor Klara wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java,
> >  line 648
> > 
> >
> > What is the reason for this null check?

In the original version when we read a lock from zookeeper we set the clinetIp 
of the lock to the current server's clientIp.
If I leave it as it is, then we could not differentiate between the lock 
created by this server, and locks created by other HS2 servers connected to 
this zookeeper instance.

I am not sure why we set clientIp at all, since it could come from the 
zookeeper server and we use it only for display purposes (at least where I 
found reference for it). That is why I raised the question in the review, and 
in the jira as well.


> On Oct. 17, 2016, 12:47 p.m., Barna Zsombor Klara wrote:
> > service/src/java/org/apache/hive/service/server/HiveServer2.java, line 671
> > 
> >
> > Any reason we are biased towards the ZooKeeperHiveLockManager? I would 
> > maybe call the method releaseStaleLocks, put it into the HiveLockManager 
> > interface and then leave it to the implementation to see if those are 
> > identified by IP and released per HS2 instance or dealt with otherwise.

The HiveLockManager could be set by HiveConf. So I treat it as a public API, 
and I do not want to change it between minor releases. It seems to me there 
should be a better solution for these kind of problems - how to handle API 
changes, but this is another topic altogether :)

The current implementations of HiveLockManager:
- ZookeeperHiveLockManager - we added the feature for that
- EmbeddedLockManager - using memory to store locks - not applicable
- DBLockManager - lock methods are throwing UnsupportedOperationException - not 
applicable


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52923/#review152862
---


On Oct. 18, 2016, 9:23 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52923/
> ---
> 
> (Updated Oct. 18, 2016, 9:23 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Marta Kuczora, Miklos Csanady, 
> namit jain, Sergio Pena, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-14979
> https://issues.apache.org/jira/browse/HIVE-14979
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding a new configuration option to HiveConf to signal whether stale lock 
> removal is requested on startup.
> Adding a new method to ZooKeeperHiveLockManager to remove stale locks
> Modifying the HiveServer2 to instantiate a lock manager and call the new 
> method if defined by the configuration.
> 
> Please take extra care when reviewing these:
> - Modifying the lock fetching method to use the clientIp from the lock, and 
> not update with the current ip - Not sure why it was done before
> - Instantiation of the lock manager - I might not chose the best method for it
> 
> Open for any suggestions :)
> 
> Thanks,
> Peter
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8ffae3b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
>  14d0ef4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 590b1f3 
> 
> Diff: https://reviews.apache.org/r/52923/diff/
> 
> 
> Testing
> ---
> 
> Created 2 unit test cases:
> - Removing own locks
> - Not removing other server's locks
> 
> Manually tested the Lock manager instantiation method on HiveServer2
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 52923: HIVE-14979 Removing stale Zookeeper locks at HiveServer2 initialization

[Peter Vary]

> On Oct. 17, 2016, 12:47 p.m., Barna Zsombor Klara wrote:
> > Thanks for the patch Peter, looks good. I only have some 
> > questions/comments, but no concrete issues to open.

Oh, and thanks for the review - useful as always :)


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52923/#review152862
---


On Oct. 18, 2016, 9:23 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52923/
> ---
> 
> (Updated Oct. 18, 2016, 9:23 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Marta Kuczora, Miklos Csanady, 
> namit jain, Sergio Pena, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-14979
> https://issues.apache.org/jira/browse/HIVE-14979
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding a new configuration option to HiveConf to signal whether stale lock 
> removal is requested on startup.
> Adding a new method to ZooKeeperHiveLockManager to remove stale locks
> Modifying the HiveServer2 to instantiate a lock manager and call the new 
> method if defined by the configuration.
> 
> Please take extra care when reviewing these:
> - Modifying the lock fetching method to use the clientIp from the lock, and 
> not update with the current ip - Not sure why it was done before
> - Instantiation of the lock manager - I might not chose the best method for it
> 
> Open for any suggestions :)
> 
> Thanks,
> Peter
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8ffae3b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
>  14d0ef4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 590b1f3 
> 
> Diff: https://reviews.apache.org/r/52923/diff/
> 
> 
> Testing
> ---
> 
> Created 2 unit test cases:
> - Removing own locks
> - Not removing other server's locks
> 
> Manually tested the Lock manager instantiation method on HiveServer2
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

[jira] [Created] (HIVE-14997) Hive query left join get wrong result

[lios.li (JIRA)]

lios.li created HIVE-14997:
--

 Summary: Hive query left join get wrong result
 Key: HIVE-14997
 URL: https://issues.apache.org/jira/browse/HIVE-14997
 Project: Hive
  Issue Type: Bug
  Components: SQL
Affects Versions: 0.14.0
 Environment: Hive 0.14.0
Subversion 
file:///Users/ghagleitner/Projects/hive-svn/rel-prep/hive-14-rel-prep -r Unknown
Compiled by ghagleitner on Sat Nov 8 23:25:06 PST 2014
>From source with checksum 49c2182a0856f7917f571802a7594b00
Reporter: lios.li


First, create two tables.

CREATE DATABASE IF NOT EXISTS test;
USE test;

DROP TABLE IF EXISTS student_info;
CREATE TABLE IF NOT EXISTS student_info(
id string COMMENT 'student id',
name string COMMENT 'student name'
)
PARTITIONED BY (l_date string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t';

ALTER TABLE test.student_info SET SERDEPROPERTIES('serialization.null.format' = 
'');

DROP TABLE IF EXISTS student_score;
CREATE TABLE IF NOT EXISTS student_score(
id string COMMENT 'student id',
class string COMMENT 'class',
score int COMMENT 'class score'
)
PARTITIONED BY (l_date string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t';

ALTER TABLE test.student_score SET SERDEPROPERTIES('serialization.null.format' 
= '');

4 records in table student_info,
1   jobs
2   cook
3   gates
4   musk

3 records in table student_score,
1   math98
2   math96
3   math94

I want get the student who has no score and id is '4'.

select * from test.student_info a
left join test.student_score b
on a.id=b.id
where (b.id='' or b.id is null)
and a.id='4';

and i got nothing.
but, i add the 'trim()'.

select * from test.student_info a
left join test.student_score b
on a.id=b.id
where (b.id='' or b.id is null)
and trim(a.id)='4';

i can get what i want.

a.ida.name  b.idb.class b.score
4   muskNULLNULLNULL

so, i think there is a bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52981: HIVE-14679: csv2/tsv2 output format disables quoting by default and it's difficult to enable

[Barna Zsombor Klara]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52981/#review153059
---



Thank you for the patch Jianguo Tian. LGTM, pending tests.
Hope you don't mind that I had a look as well.


beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java (line 559)


nit: extra spaces



beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java (line 567)


nit: extra spaces



beeline/src/java/org/apache/hive/beeline/SeparatedValuesOutputFormat.java (line 
104)


I think we can remove this line, not just comment it out. (Same for line 
36).



beeline/src/main/resources/BeeLine.properties (line 190)


Missing verb in the sentence.
will not be visible around a value? or will not be present around a value?


- Barna Zsombor Klara


On Oct. 18, 2016, 8:11 a.m., Jianguo Tian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52981/
> ---
> 
> (Updated Oct. 18, 2016, 8:11 a.m.)
> 
> 
> Review request for hive and cheng xu.
> 
> 
> Bugs: HIVE-14679
> https://issues.apache.org/jira/browse/HIVE-14679
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1. Quoting should be enabled by default for csv2, tsv2 and dsv.
> 2. Disabling quoting should be possible using a beeline argument.
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 79922d2 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 57b9c46 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 039e354 
>   beeline/src/java/org/apache/hive/beeline/SeparatedValuesOutputFormat.java 
> 66d9fd0 
>   beeline/src/main/resources/BeeLine.properties ad79c01 
> 
> Diff: https://reviews.apache.org/r/52981/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jianguo Tian
> 
>

Re: Review Request 52923: HIVE-14979 Removing stale Zookeeper locks at HiveServer2 initialization

[Barna Zsombor Klara]

> On Oct. 17, 2016, 12:47 p.m., Barna Zsombor Klara wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java,
> >  line 648
> > 
> >
> > What is the reason for this null check?
> 
> Peter Vary wrote:
> In the original version when we read a lock from zookeeper we set the 
> clinetIp of the lock to the current server's clientIp.
> If I leave it as it is, then we could not differentiate between the lock 
> created by this server, and locks created by other HS2 servers connected to 
> this zookeeper instance.
> 
> I am not sure why we set clientIp at all, since it could come from the 
> zookeeper server and we use it only for display purposes (at least where I 
> found reference for it). That is why I raised the question in the review, and 
> in the jira as well.

I see, thanks for the clarification. Maybe this could be mentioned in a 
comment, since it is not obvious at first glance.


> On Oct. 17, 2016, 12:47 p.m., Barna Zsombor Klara wrote:
> > service/src/java/org/apache/hive/service/server/HiveServer2.java, line 671
> > 
> >
> > Any reason we are biased towards the ZooKeeperHiveLockManager? I would 
> > maybe call the method releaseStaleLocks, put it into the HiveLockManager 
> > interface and then leave it to the implementation to see if those are 
> > identified by IP and released per HS2 instance or dealt with otherwise.
> 
> Peter Vary wrote:
> The HiveLockManager could be set by HiveConf. So I treat it as a public 
> API, and I do not want to change it between minor releases. It seems to me 
> there should be a better solution for these kind of problems - how to handle 
> API changes, but this is another topic altogether :)
> 
> The current implementations of HiveLockManager:
> - ZookeeperHiveLockManager - we added the feature for that
> - EmbeddedLockManager - using memory to store locks - not applicable
> - DBLockManager - lock methods are throwing UnsupportedOperationException 
> - not applicable

OK, didn't know the scope of the fix.
However I would argue against that DBLockManager is not applicable. While some 
lock methods are throwing unsupported op exceptions, it has a package private 
lock method called from the DbTxnManager#acquireLocks.
Also adding a method to the HiveLockManager interface would have the added 
benefit that future implementations may be able to decide on their own if this 
needs handling or not.
But I'll leave it to you (and other reviewers) to decide.


- Barna Zsombor


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52923/#review152862
---


On Oct. 18, 2016, 9:23 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52923/
> ---
> 
> (Updated Oct. 18, 2016, 9:23 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Marta Kuczora, Miklos Csanady, 
> namit jain, Sergio Pena, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-14979
> https://issues.apache.org/jira/browse/HIVE-14979
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding a new configuration option to HiveConf to signal whether stale lock 
> removal is requested on startup.
> Adding a new method to ZooKeeperHiveLockManager to remove stale locks
> Modifying the HiveServer2 to instantiate a lock manager and call the new 
> method if defined by the configuration.
> 
> Please take extra care when reviewing these:
> - Modifying the lock fetching method to use the clientIp from the lock, and 
> not update with the current ip - Not sure why it was done before
> - Instantiation of the lock manager - I might not chose the best method for it
> 
> Open for any suggestions :)
> 
> Thanks,
> Peter
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8ffae3b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
>  14d0ef4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 590b1f3 
> 
> Diff: https://reviews.apache.org/r/52923/diff/
> 
> 
> Testing
> ---
> 
> Created 2 unit test cases:
> - Removing own locks
> - Not removing other server's locks
> 
> Manually tested the Lock manager instantiation method on HiveServer2
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Start HiveServer2 with Kerberos meet FATAL error

[Vivek Shrivastava]

What is the output of
klist -ket /home/work/software/hive/conf/hive.keytab
and
hostname
and
hostname -f

On Tue, Oct 18, 2016 at 4:18 AM, Micro dong  wrote:

> tempt48  is the local machine's hostname
> which is configured in /etc/hosts.
>
> 2016-10-18 15:33 GMT+08:00 Chen, Junjie J :
>
> > Please try to use full qualified name of host,  such as hive2/
> > tempt48.hadoop@hadoop.com
> >
> > -Original Message-
> > From: Micro dong [mailto:microle.d...@gmail.com]
> > Sent: Tuesday, October 18, 2016 2:37 PM
> > To: dev@hive.apache.org
> > Subject: Re: Start HiveServer2 with Kerberos meet FATAL error
> >
> > I follow your suggestion and it also meet the same error too.
> >
> > *start hiveservice*
> > /home/work/software/hive/bin/hive --service hiveserver2
> >
> >
> > *klist output*
> > Ticket cache: FILE:/tmp/krb5cc_31100
> > Default principal: hive2/temp...@hadoop.com
> >
> > Valid starting ExpiresService principal
> > 10/18/16 14:23:04  10/19/16 14:23:04  krbtgt/hadoop@hadoop.com renew
> > until 10/28/16 14:23:04
> >
> > 2016-10-18 12:07 GMT+08:00 Vivek Shrivastava :
> >
> > > can you please change this line
> > >
> > > hive2/_h...@hadoop.com * with
> > > hive2/_h...@hadoop.com*
> > >
> > > How are you starting the service, What is the klist output?
> > >
> > >
> > > On Mon, Oct 17, 2016 at 11:09 PM, Micro dong 
> > > wrote:
> > >
> > > > I'm trying to configure HiveServer2(hive-1.2.1)  With Kerberos。Here
> > > > is my Hive's configuration file。
> > > >  **
> > > > *hive.server2.authentication*
> > > > *KERBEROS*
> > > > *  *
> > > > *  *
> > > > *hive.server2.authentication.kerberos.principal*
> > > > *hive2/_h...@hadoop.com *
> > > > *  *
> > > > *  *
> > > > *hive.server2.authentication.kerberos.keytab*
> > > > */home/work/software/hive/conf/hive.keytab*
> > > > *  *
> > > >
> > > > the keytab file is in its location, its owner is work. But when I
> > > > try to start the HiveServer2, I see this message on the log:
> > > >
> > > > *2016-10-18 10:20:24,867 FATAL [Thread-9]: thrift.ThriftCLIService
> > > > (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2:
> > > could
> > > > not start ThriftBinaryCLIService*
> > > > *javax.security.auth.login.LoginException: Kerberos principal should
> > > have
> > > > 3
> > > > parts: work**  at *
> > > > *org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(
> > > > HiveAuthFactory.java:147)*
> > > > *atorg.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(
> > > > ThriftBinaryCLIService.java:58)*
> > > > *at java.lang.Thread.run(Thread.java:722)*
> > > >
> > > > Here *work* is my unix login name. Any help would be highly
> > appreciated.
> > > > Best regards
> > > >
> > >
> >
> >
> >
> > --
> > 英者自知，雄者自胜。
> >
>
>
>
> --
> 英者自知，雄者自胜。
>

[jira] [Created] (HIVE-14998) TestPluggableHiveSessionImpl doesn't test anything

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-14998:
---

 Summary: TestPluggableHiveSessionImpl doesn't test anything
 Key: HIVE-14998
 URL: https://issues.apache.org/jira/browse/HIVE-14998
 Project: Hive
  Issue Type: Bug
  Components: Tests
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


this test either prints an exception to the stdout ... or not - in its current 
form it doesn't really usefull.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 52984: HIVE-14391 TestAccumuloCliDriver is not executed during precommit tests

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52984/
---

Review request for hive, Zoltan Haindrich, Marta Kuczora, Miklos Csanady, 
Sergio Pena, Siddharth Seth, and Barna Zsombor Klara.


Bugs: HIVE-14391
https://issues.apache.org/jira/browse/HIVE-14391


Repository: hive-git


Description
---

The aim of this patch, to enable the Accumulo qtests to run again.

It contains the following:
- AccumuloQTestUtil change, so the configuration changes are set not only in 
init(), but in clearTestSideEffects() as well
- AccumuloTestSetup changes:
   - Removing temporary
   - Configuration update
- 3 qtest golden output rewrites - here I am a little nervous, since I am not 
sure that all of the change are caused by other changes in Hive - I would be 
happy if someone with more experience can confirm it.
- master-mr2.properties change to run the tests


Diffs
-

  accumulo-handler/src/test/results/positive/accumulo_predicate_pushdown.q.out 
eb03041 
  accumulo-handler/src/test/results/positive/accumulo_queries.q.out d3c717b 
  
accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
 07af080 
  itests/qtest-accumulo/pom.xml a4a5c75 
  
itests/util/src/main/java/org/apache/hadoop/hive/accumulo/AccumuloQTestUtil.java
 88bc0bc 
  
itests/util/src/main/java/org/apache/hadoop/hive/accumulo/AccumuloTestSetup.java
 73d5f15 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreAccumuloCliDriver.java
 a5d2711 
  testutils/ptest2/conf/deployed/master-mr2.properties 02a560e 

Diff: https://reviews.apache.org/r/52984/diff/


Testing
---

Run the test on my computer several times in several combination


Thanks,

Peter Vary

[jira] [Created] (HIVE-14999) SparkClientUtilities does not support viewFS

[Teruyoshi Zenmyo (JIRA)]

Teruyoshi Zenmyo created HIVE-14999:
---

 Summary: SparkClientUtilities does not support viewFS
 Key: HIVE-14999
 URL: https://issues.apache.org/jira/browse/HIVE-14999
 Project: Hive
  Issue Type: Bug
  Components: Spark
Reporter: Teruyoshi Zenmyo


In SparkClientUtilities.urlFromPathString(), viewFS is not considered and jars 
on viewFS cannot be found.
This would cause a number of problems, for instance, custom UDFs are not 
available on viewFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15000) Remove addlocaldriverjar, and addlocaldrivername from command line help

[Peter Vary (JIRA)]

Peter Vary created HIVE-15000:
-

 Summary: Remove addlocaldriverjar, and addlocaldrivername from 
command line help
 Key: HIVE-15000
 URL: https://issues.apache.org/jira/browse/HIVE-15000
 Project: Hive
  Issue Type: Sub-task
Reporter: Peter Vary


As discussed with [~Ferd], the following commands are not working, and never 
were intended as a command line parameters, so they should be removed from the 
command line help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15001) Remove showConnectedUrl from command line help

[Peter Vary (JIRA)]

Peter Vary created HIVE-15001:
-

 Summary: Remove showConnectedUrl from command line help
 Key: HIVE-15001
 URL: https://issues.apache.org/jira/browse/HIVE-15001
 Project: Hive
  Issue Type: Sub-task
  Components: Beeline
Reporter: Peter Vary
Assignee: Peter Vary
Priority: Trivial


As discussed with [~nemon], the showConnectedUrl commandline parameter is not 
working since a erroneous merge. Instead beeline always prints the currently 
connected url. Since it is good for everyone, no extra parameter is needed to 
turn this feature on.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15002) HiveSessionImpl#executeStatementInternal may leave locks in an inconsistent state

[Zoltan Haindrich (JIRA)]

Zoltan Haindrich created HIVE-15002:
---

 Summary: HiveSessionImpl#executeStatementInternal may leave locks 
in an inconsistent state
 Key: HIVE-15002
 URL: https://issues.apache.org/jira/browse/HIVE-15002
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


{{TestEmbeddedThriftBinaryCLIService}} have showed me this behaviour.

Thrift service may hang because of this issue.
{{TestEmbeddedThriftBinaryCLIService}} can reproduce this problem with a very 
bad probability...but I was able to 

Its not able to aquire the semaphore; because most probably because an earlier 
command have produced an SQLException in the unprotected parts.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52981: HIVE-14679: csv2/tsv2 output format disables quoting by default and it's difficult to enable

[Jianguo Tian]

> On Oct. 18, 2016, 9:30 a.m., Barna Zsombor Klara wrote:
> > Thank you for the patch Jianguo Tian. LGTM, pending tests.
> > Hope you don't mind that I had a look as well.

Really appreciate for your review. I'll fix these problems and add tests.


- Jianguo


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52981/#review153059
---


On Oct. 18, 2016, 8:11 a.m., Jianguo Tian wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52981/
> ---
> 
> (Updated Oct. 18, 2016, 8:11 a.m.)
> 
> 
> Review request for hive and cheng xu.
> 
> 
> Bugs: HIVE-14679
> https://issues.apache.org/jira/browse/HIVE-14679
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1. Quoting should be enabled by default for csv2, tsv2 and dsv.
> 2. Disabling quoting should be possible using a beeline argument.
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 79922d2 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 57b9c46 
>   beeline/src/java/org/apache/hive/beeline/Commands.java 039e354 
>   beeline/src/java/org/apache/hive/beeline/SeparatedValuesOutputFormat.java 
> 66d9fd0 
>   beeline/src/main/resources/BeeLine.properties ad79c01 
> 
> Diff: https://reviews.apache.org/r/52981/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jianguo Tian
> 
>

Re: Review Request 52978: HIVE-14459 TestBeeLineDriver - migration and re-enable

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52978/
---

(Updated Oct. 18, 2016, 2:54 p.m.)


Review request for hive, Zoltan Haindrich, Marta Kuczora, Miklos Csanady, 
Prasanth_J, Sergey Shelukhin, Sergio Pena, Siddharth Seth, and Barna Zsombor 
Klara.


Changes
---

Updated the diff algorithm, so the diff results are printed not just an 
unhelpfull yes/no.
Hidden an asynchronous log message.


Bugs: HIVE-14459
https://issues.apache.org/jira/browse/HIVE-14459


Repository: hive-git


Description
---

Kept the changes minimal with the sole goal to be able to run the BeeLine query 
tests multiple times successfully.
- Enabled the driver
- Modified the regexps to hide when comparing the results
- Configured to run only 1 qtest file - so we can test, and could decide later 
of the beeline testing scope
- Added required dependencies to pom
- Added specific results dir for beeline q.out-s


Diffs (updated)
-

  beeline/src/java/org/apache/hive/beeline/util/QFileClient.java 81f1b0e 
  
itests/qtest/src/test/java/org/apache/hadoop/hive/cli/DisabledTestBeeLineDriver.java
 cb276e6 
  itests/qtest/src/test/java/org/apache/hadoop/hive/cli/TestBeeLineDriver.java 
PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 97e310d 
  itests/util/pom.xml 6d93dc1 
  itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CliConfigs.java 
366c7b4 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
 e5144e3 
  ql/src/test/results/clientpositive/beeline/escape_comments.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/52978/diff/


Testing
---

Manually on my computer several runs.
Waiting for the QA tests


Thanks,

Peter Vary

Re: Review Request 52427: HIVE-14580 - Introduce || operator

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52427/
---

(Updated Oct. 18, 2016, 3:06 p.m.)


Review request for hive, Ashutosh Chauhan and pengcheng xiong.


Changes
---

parser-only change ;)


Bugs: HIVE-14580
https://issues.apache.org/jira/browse/HIVE-14580


Repository: hive-git


Description
---

adds support for || text concatenation

* reuses the concat udf for doing the real work
* same precedence as + / - ; I think this is expected
* small ast rewrite code in SemanticAnalyzer
* details about other db || operators: 
https://issues.apache.org/jira/browse/HIVE-14580?focusedCommentId=15546578&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15546578


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 
5d3fa6aeea5251e461c4b0a8277b4ff1b55f9115 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 
50987c391e435f4e3c71243e90af89fc52f380aa 
  ql/src/test/queries/clientpositive/concat_op.q PRE-CREATION 
  ql/src/test/results/clientpositive/concat_op.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/52427/diff/


Testing
---

added qtest


Thanks,

Zoltan Haindrich

Re: Review Request 52427: HIVE-14580 - Introduce || operator

[Zoltan Haindrich]

> On Oct. 18, 2016, 6:51 a.m., pengcheng xiong wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g, line 445
> > 
> >
> > change this to
> > {code}
> > precedenceConcatenateExpression
> > :
> > (precedencePlusExpression -> precedencePlusExpression)
> > (
> > precedenceConcatenateOperator plus=precedencePlusExpression
> > -> ^(TOK_FUNCTION {adaptor.create(Identifier, "concat")} 
> > {$precedenceConcatenateExpression.tree} $plus)
> > )*
> > -> {$precedenceConcatenateExpression.tree}
> > ;
> > {code}
> > So that we can finish the rewriting in the parser.

i've experimented with the same thing...but I was unable to put it together ;)

thank you for showing it, its far better than that ugly ast rewrite...


> On Oct. 18, 2016, 6:51 a.m., pengcheng xiong wrote:
> > ql/src/test/queries/clientpositive/concat_op.q, line 3
> > 
> >
> > Please confirm that Postgres and/Oracle returns the same result.

If the question is about and/or precedence vs other databases: then yes, hive 
is consistent with them.

I've added a more complete check of the same thing at the end of the qfile - I 
just wanted to check it; and didn't notice it being left in the qfile...but it 
should never break...so I guess its better to add it.


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52427/#review153040
---


On Oct. 18, 2016, 3:06 p.m., Zoltan Haindrich wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52427/
> ---
> 
> (Updated Oct. 18, 2016, 3:06 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and pengcheng xiong.
> 
> 
> Bugs: HIVE-14580
> https://issues.apache.org/jira/browse/HIVE-14580
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> adds support for || text concatenation
> 
> * reuses the concat udf for doing the real work
> * same precedence as + / - ; I think this is expected
> * small ast rewrite code in SemanticAnalyzer
> * details about other db || operators: 
> https://issues.apache.org/jira/browse/HIVE-14580?focusedCommentId=15546578&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15546578
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 
> 5d3fa6aeea5251e461c4b0a8277b4ff1b55f9115 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 
> 50987c391e435f4e3c71243e90af89fc52f380aa 
>   ql/src/test/queries/clientpositive/concat_op.q PRE-CREATION 
>   ql/src/test/results/clientpositive/concat_op.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/52427/diff/
> 
> 
> Testing
> ---
> 
> added qtest
> 
> 
> Thanks,
> 
> Zoltan Haindrich
> 
>

Re: Review Request 51755: Support Intersect Distinct

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51755/#review153098
---




itests/src/test/resources/testconfiguration.properties (line 53)


Lets do it other way round. Write all new tests on MiniLlapLocalDriver and 
have one on MR.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 
(lines 130 - 132)


Is this needed?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitPullUpConstantsRule.java
 (lines 162 - 170)


Now this is covered in different jira, can be removed.



ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 1528)


Add opcode in error message.



ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java
 (line 41)


Any reason for adding Minus as reserved word?



ql/src/test/results/clientpositive/auto_join8.q.out (line 94)


Not folding null constant anymore?



ql/src/test/results/clientpositive/input25.q.out (line 122)


Shuffling unnecssary constant column?



ql/src/test/results/clientpositive/join8.q.out (line 97)


Not folding null constant anymore?



ql/src/test/results/clientpositive/smb_mapjoin_25.q.out (line 116)


Shuffling unnecessary column?


- Ashutosh Chauhan


On Oct. 15, 2016, 4:10 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51755/
> ---
> 
> (Updated Oct. 15, 2016, 4:10 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12765
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 97e310d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 
> eb0ba7b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> c527e58 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> cf93ed8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExcept.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectMergeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitPullUpConstantsRule.java
>  cc318db 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  8d738aa 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6ab947 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 5d3fa6a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 5c16c55 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 50987c3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBExpr.java cccf0f6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 747f387 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java
>  a427803 
>   ql/src/test/queries/clientpositive/except_all.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/except_distinct.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_all.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_distinct.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_merge.q PRE-CREATION 
>   ql/src/test/results/clientpositive/auto_join8.q.out ccbafba 
>   ql/src/test/results/clientpositive/except_all.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/except_distinct.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/input25.q.out 465173d 
>   ql/src/test/results/clientpositive/intersect_all.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/intersect_distinct.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/join8.q.out 47b821a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> 45f2481 
>   ql/src/test/results/clientpositive/smb_mapjoin_25.q.out aba

Re: Review Request 51755: Support Intersect Distinct

[Ashutosh Chauhan]

> On Oct. 7, 2016, 2:57 a.m., Ashutosh Chauhan wrote:
> >

Took first pass look. You may wait to address these comments till I finish full 
review.


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51755/#review148788
---


On Oct. 15, 2016, 4:10 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51755/
> ---
> 
> (Updated Oct. 15, 2016, 4:10 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12765
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 97e310d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 
> eb0ba7b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> c527e58 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> cf93ed8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExcept.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectMergeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitPullUpConstantsRule.java
>  cc318db 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  8d738aa 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6ab947 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 5d3fa6a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 5c16c55 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 50987c3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBExpr.java cccf0f6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 747f387 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java
>  a427803 
>   ql/src/test/queries/clientpositive/except_all.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/except_distinct.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_all.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_distinct.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_merge.q PRE-CREATION 
>   ql/src/test/results/clientpositive/auto_join8.q.out ccbafba 
>   ql/src/test/results/clientpositive/except_all.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/except_distinct.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/input25.q.out 465173d 
>   ql/src/test/results/clientpositive/intersect_all.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/intersect_distinct.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/join8.q.out 47b821a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> 45f2481 
>   ql/src/test/results/clientpositive/smb_mapjoin_25.q.out aba899e 
>   ql/src/test/results/clientpositive/spark/auto_join8.q.out 6909113 
>   ql/src/test/results/clientpositive/spark/join8.q.out 4903c90 
>   ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out eacb438 
>   ql/src/test/results/clientpositive/spark/union_top_level.q.out e1c7fc7 
>   ql/src/test/results/clientpositive/tez/intersect_merge.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/timestamp.q.out 7c08ec8 
>   ql/src/test/results/clientpositive/vector_null_projection.q.out 779f787 
> 
> Diff: https://reviews.apache.org/r/51755/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 52708: HIVE-14913 Moved system tests to unit tests

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52708/
---

(Updated Oct. 18, 2016, 5:46 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-14913
https://issues.apache.org/jira/browse/HIVE-14913


Repository: hive-git


Description
---

Moved bunch of system tests to hive unit tests


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 97e310d 
  ql/src/test/queries/clientpositive/cte_1.q ca4132f 
  ql/src/test/queries/clientpositive/cte_2.q b49620a 
  ql/src/test/queries/clientpositive/current_date_timestamp.q 9bed885 
  ql/src/test/queries/clientpositive/insert1.q 4117b54 
  ql/src/test/queries/clientpositive/join_acid_non_acid.q 866b466 
  ql/src/test/queries/clientpositive/lvj_mapjoin.q b726e2a 
  ql/src/test/queries/clientpositive/orc_ppd_basic.q 43f2c85 
  ql/src/test/queries/clientpositive/schema_evol_orc_acid_part_update.q d971649 
  ql/src/test/queries/clientpositive/vectorization_0.q caa6a6a 
  ql/src/test/results/clientpositive/cte_1.q.out 6e7acf6 
  ql/src/test/results/clientpositive/cte_2.q.out 4b0bc71 
  ql/src/test/results/clientpositive/current_date_timestamp.q.out 9c26095 
  ql/src/test/results/clientpositive/insert1.q.out 7a2c429 
  ql/src/test/results/clientpositive/llap/cte_1.q.out 41ffdc7 
  ql/src/test/results/clientpositive/llap/cte_2.q.out 017bded 
  ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/insert1.q.out 4b30156 
  ql/src/test/results/clientpositive/llap/join_acid_non_acid.q.out 4905351 
  ql/src/test/results/clientpositive/llap/lvj_mapjoin.q.out dd4d024 
  ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out aa37ff3 
  
ql/src/test/results/clientpositive/llap/schema_evol_orc_acid_part_update.q.out 
ed698e1 
  ql/src/test/results/clientpositive/llap/vectorization_0.q.out faceb5c 
  ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out b5a6d04 
  ql/src/test/results/clientpositive/spark/vectorization_0.q.out 22fe7cd 
  ql/src/test/results/clientpositive/vectorization_0.q.out ac33721 

Diff: https://reviews.apache.org/r/52708/diff/


Testing
---

Pre-commit testing


Thanks,

Vineet Garg

Re: Review Request 52708: HIVE-14913 Moved system tests to unit tests

[Vineet Garg]

> On Oct. 14, 2016, 8:23 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/insert1.q, lines 49-51
> > 
> >
> > insert overwrite local dir is best tested on cluster. So, doesnt make 
> > sense in ut.

So should we keep this in system tests ?


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52708/#review152425
---


On Oct. 18, 2016, 5:46 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52708/
> ---
> 
> (Updated Oct. 18, 2016, 5:46 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-14913
> https://issues.apache.org/jira/browse/HIVE-14913
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Moved bunch of system tests to hive unit tests
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 97e310d 
>   ql/src/test/queries/clientpositive/cte_1.q ca4132f 
>   ql/src/test/queries/clientpositive/cte_2.q b49620a 
>   ql/src/test/queries/clientpositive/current_date_timestamp.q 9bed885 
>   ql/src/test/queries/clientpositive/insert1.q 4117b54 
>   ql/src/test/queries/clientpositive/join_acid_non_acid.q 866b466 
>   ql/src/test/queries/clientpositive/lvj_mapjoin.q b726e2a 
>   ql/src/test/queries/clientpositive/orc_ppd_basic.q 43f2c85 
>   ql/src/test/queries/clientpositive/schema_evol_orc_acid_part_update.q 
> d971649 
>   ql/src/test/queries/clientpositive/vectorization_0.q caa6a6a 
>   ql/src/test/results/clientpositive/cte_1.q.out 6e7acf6 
>   ql/src/test/results/clientpositive/cte_2.q.out 4b0bc71 
>   ql/src/test/results/clientpositive/current_date_timestamp.q.out 9c26095 
>   ql/src/test/results/clientpositive/insert1.q.out 7a2c429 
>   ql/src/test/results/clientpositive/llap/cte_1.q.out 41ffdc7 
>   ql/src/test/results/clientpositive/llap/cte_2.q.out 017bded 
>   ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/insert1.q.out 4b30156 
>   ql/src/test/results/clientpositive/llap/join_acid_non_acid.q.out 4905351 
>   ql/src/test/results/clientpositive/llap/lvj_mapjoin.q.out dd4d024 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out aa37ff3 
>   
> ql/src/test/results/clientpositive/llap/schema_evol_orc_acid_part_update.q.out
>  ed698e1 
>   ql/src/test/results/clientpositive/llap/vectorization_0.q.out faceb5c 
>   ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out b5a6d04 
>   ql/src/test/results/clientpositive/spark/vectorization_0.q.out 22fe7cd 
>   ql/src/test/results/clientpositive/vectorization_0.q.out ac33721 
> 
> Diff: https://reviews.apache.org/r/52708/diff/
> 
> 
> Testing
> ---
> 
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

[jira] [Created] (HIVE-15003) Update 'ALTER TABLE...UPDATE STATISTICS FOR COLUMN..' statement to support more data types

[Vineet Garg (JIRA)]

Vineet Garg created HIVE-15003:
--

 Summary: Update 'ALTER TABLE...UPDATE STATISTICS FOR COLUMN..' 
statement to support more data types
 Key: HIVE-15003
 URL: https://issues.apache.org/jira/browse/HIVE-15003
 Project: Hive
  Issue Type: Task
  Components: Statistics
Reporter: Vineet Garg
Assignee: Vineet Garg


Currently ALTER TABLE...UDPATE STATISTICS FOR COLUMN... only support updating 
statistics for following data types:
* STRING
* DOUBLE
* BOOLEAN
* BINARY
* DECIMAL
* DATE

We are missing the following data types
* TINYINT
* SMALLINT
* INT
* BIGINT
* FLOAT
* VARCHAR
* CHAR
* TIMESTAMP



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15004) Query for parquet tables failing with java.lang.IllegalArgumentException: FilterPredicate column: f's declared type (java.lang.Double) does not match the schema found in

[Aswathy Chellammal Sreekumar (JIRA)]

Aswathy Chellammal Sreekumar created HIVE-15004:
---

 Summary: Query for parquet tables failing with 
java.lang.IllegalArgumentException: FilterPredicate column: f's declared type 
(java.lang.Double) does not match the schema found in file metadata. 
 Key: HIVE-15004
 URL: https://issues.apache.org/jira/browse/HIVE-15004
 Project: Hive
  Issue Type: Bug
  Components: Hive, HiveServer2
Affects Versions: 1.2.1
Reporter: Aswathy Chellammal Sreekumar
Assignee: Jason Dere
 Fix For: 1.2.1


Queries involving float data type, run against parquet tables failing

hive> desc extended all100k;
OK
t     tinyint
sismallint
i     int
b     bigint
f     float
d     double
s     string
dcdecimal(38,18)
boboolean
v     varchar(25)
c     char(25)
tstimestamp
dtdate

Detailed Table Information  Table(tableName:all100k, dbName:default, 
owner:hrt_qa, createTime:1476765150, lastAccessTime:0, retention:0, 
sd:StorageDescriptor(cols:[FieldSchema(name:t, type:tinyint, comment:null), 
FieldSchema(name:si, type:smallint, comment:null), FieldSchema(name:i, 
type:int, comment:null), FieldSchema(name:b, type:bigint, comment:null), 
FieldSchema(name:f, type:float, comment:null), FieldSchema(name:d, type:double, 
comment:null), FieldSchema(name:s, type:string, comment:null), 
FieldSchema(name:dc, type:decimal(38,18), comment:null), FieldSchema(name:bo, 
type:boolean, comment:null), FieldSchema(name:v, type:varchar(25), 
comment:null), FieldSchema(name:c, type:char(25), comment:null), 
FieldSchema(name:ts, type:timestamp, comment:null), FieldSchema(name:dt, 
type:date, comment:null)], 
location:hdfs://ctr-e45-1475874954070-9012-01-08.hwx.site:8020/apps/hive/warehouse/all100k,
 inputFormat:org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat, 
outputFormat:org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat, 
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, 
serializationLib:org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe, 
parameters:{serialization.format=1}), bucketCols:[], sortCols:[], 
parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], 
skewedColValueLocationMaps:{}), storedAsSubDirectories:false), 
partitionKeys:[], parameters:{numFiles=1, transient_lastDdlTime=1476765184, 
COLUMN_STATS_ACCURATE={"COLUMN_STATS":{"t":"true","si":"true","i":"true","b":"true","f":"true","d":"true","s":"true","dc":"true","bo":"true","v":"true","c":"true","ts":"true"},"BASIC_STATS":"true"},
 totalSize=6564143, numRows=10, rawDataSize=130}, 
viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
Time taken: 0.54 seconds, Fetched: 15 row(s)
hive> select t from all100k
> where t<>0 and s<>0 and b<>0 and (f<>0 or d<>0);
OK
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
details.
Failed with exception java.io.IOException:java.lang.IllegalArgumentException: 
FilterPredicate column: f's declared type (java.lang.Double) does not match the 
schema found in file metadata. Column f is of type: FLOAT
Valid types for this column are: [class java.lang.Float]
Time taken: 0.919 seconds



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15005) Path does not add slash in hive.cluster.delegation.token.store.zookeeper.znode config

[Huizhi Lu (JIRA)]

Huizhi Lu created HIVE-15005:


 Summary: Path does not add slash in 
hive.cluster.delegation.token.store.zookeeper.znode config
 Key: HIVE-15005
 URL: https://issues.apache.org/jira/browse/HIVE-15005
 Project: Hive
  Issue Type: Bug
Reporter: Huizhi Lu
Assignee: Huizhi Lu


If set 
'hive.cluster.delegation.token.store.zookeeper.znode=/hive/cluster/delegation',
the new node path will be '/hive/cluster/delegationMETASTORE/tokens'.

It should be /hive/cluster/delegation/METASTORE/tokens.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52708: HIVE-14913 Moved system tests to unit tests

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52708/#review153119
---




ql/src/test/queries/clientpositive/acid_globallimit.q (line 16)


Worth keeping this second query.



ql/src/test/queries/clientpositive/orc_ppd_basic.q (lines 214 - 219)


Aren't these tests repeated?



ql/src/test/queries/clientpositive/orc_ppd_basic.q (lines 237 - 239)


Repeated tests.


- Ashutosh Chauhan


On Oct. 18, 2016, 5:46 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52708/
> ---
> 
> (Updated Oct. 18, 2016, 5:46 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-14913
> https://issues.apache.org/jira/browse/HIVE-14913
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Moved bunch of system tests to hive unit tests
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 97e310d 
>   ql/src/test/queries/clientpositive/cte_1.q ca4132f 
>   ql/src/test/queries/clientpositive/cte_2.q b49620a 
>   ql/src/test/queries/clientpositive/current_date_timestamp.q 9bed885 
>   ql/src/test/queries/clientpositive/insert1.q 4117b54 
>   ql/src/test/queries/clientpositive/join_acid_non_acid.q 866b466 
>   ql/src/test/queries/clientpositive/lvj_mapjoin.q b726e2a 
>   ql/src/test/queries/clientpositive/orc_ppd_basic.q 43f2c85 
>   ql/src/test/queries/clientpositive/schema_evol_orc_acid_part_update.q 
> d971649 
>   ql/src/test/queries/clientpositive/vectorization_0.q caa6a6a 
>   ql/src/test/results/clientpositive/cte_1.q.out 6e7acf6 
>   ql/src/test/results/clientpositive/cte_2.q.out 4b0bc71 
>   ql/src/test/results/clientpositive/current_date_timestamp.q.out 9c26095 
>   ql/src/test/results/clientpositive/insert1.q.out 7a2c429 
>   ql/src/test/results/clientpositive/llap/cte_1.q.out 41ffdc7 
>   ql/src/test/results/clientpositive/llap/cte_2.q.out 017bded 
>   ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/insert1.q.out 4b30156 
>   ql/src/test/results/clientpositive/llap/join_acid_non_acid.q.out 4905351 
>   ql/src/test/results/clientpositive/llap/lvj_mapjoin.q.out dd4d024 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out aa37ff3 
>   
> ql/src/test/results/clientpositive/llap/schema_evol_orc_acid_part_update.q.out
>  ed698e1 
>   ql/src/test/results/clientpositive/llap/vectorization_0.q.out faceb5c 
>   ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out b5a6d04 
>   ql/src/test/results/clientpositive/spark/vectorization_0.q.out 22fe7cd 
>   ql/src/test/results/clientpositive/vectorization_0.q.out ac33721 
> 
> Diff: https://reviews.apache.org/r/52708/diff/
> 
> 
> Testing
> ---
> 
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

Re: Review Request 52708: HIVE-14913 Moved system tests to unit tests

[Ashutosh Chauhan]

> On Oct. 14, 2016, 8:23 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/insert1.q, lines 49-51
> > 
> >
> > insert overwrite local dir is best tested on cluster. So, doesnt make 
> > sense in ut.
> 
> Vineet Garg wrote:
> So should we keep this in system tests ?

yes


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52708/#review152425
---


On Oct. 18, 2016, 5:46 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52708/
> ---
> 
> (Updated Oct. 18, 2016, 5:46 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-14913
> https://issues.apache.org/jira/browse/HIVE-14913
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Moved bunch of system tests to hive unit tests
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 97e310d 
>   ql/src/test/queries/clientpositive/cte_1.q ca4132f 
>   ql/src/test/queries/clientpositive/cte_2.q b49620a 
>   ql/src/test/queries/clientpositive/current_date_timestamp.q 9bed885 
>   ql/src/test/queries/clientpositive/insert1.q 4117b54 
>   ql/src/test/queries/clientpositive/join_acid_non_acid.q 866b466 
>   ql/src/test/queries/clientpositive/lvj_mapjoin.q b726e2a 
>   ql/src/test/queries/clientpositive/orc_ppd_basic.q 43f2c85 
>   ql/src/test/queries/clientpositive/schema_evol_orc_acid_part_update.q 
> d971649 
>   ql/src/test/queries/clientpositive/vectorization_0.q caa6a6a 
>   ql/src/test/results/clientpositive/cte_1.q.out 6e7acf6 
>   ql/src/test/results/clientpositive/cte_2.q.out 4b0bc71 
>   ql/src/test/results/clientpositive/current_date_timestamp.q.out 9c26095 
>   ql/src/test/results/clientpositive/insert1.q.out 7a2c429 
>   ql/src/test/results/clientpositive/llap/cte_1.q.out 41ffdc7 
>   ql/src/test/results/clientpositive/llap/cte_2.q.out 017bded 
>   ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/insert1.q.out 4b30156 
>   ql/src/test/results/clientpositive/llap/join_acid_non_acid.q.out 4905351 
>   ql/src/test/results/clientpositive/llap/lvj_mapjoin.q.out dd4d024 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out aa37ff3 
>   
> ql/src/test/results/clientpositive/llap/schema_evol_orc_acid_part_update.q.out
>  ed698e1 
>   ql/src/test/results/clientpositive/llap/vectorization_0.q.out faceb5c 
>   ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out b5a6d04 
>   ql/src/test/results/clientpositive/spark/vectorization_0.q.out 22fe7cd 
>   ql/src/test/results/clientpositive/vectorization_0.q.out ac33721 
> 
> Diff: https://reviews.apache.org/r/52708/diff/
> 
> 
> Testing
> ---
> 
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

[jira] [Created] (HIVE-15006) Flaky test: TestBeelineWithHS2ConnectionFile

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15006:
-

 Summary: Flaky test: TestBeelineWithHS2ConnectionFile
 Key: HIVE-15006
 URL: https://issues.apache.org/jira/browse/HIVE-15006
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth


Seems to time out fairly often.

https://issues.apache.org/jira/browse/HIVE-14391, 
https://builds.apache.org/job/PreCommit-HIVE-Build/1621/testReport

https://issues.apache.org/jira/browse/HIVE-14887, 
https://builds.apache.org/job/PreCommit-HIVE-Build/1606/testReport



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 52923: HIVE-14979 Removing stale Zookeeper locks at HiveServer2 initialization

[Lefty Leverenz]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52923/#review153057
---




common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (lines 1766 - 1772)


1. camel caps "zookeeper" -> "ZooKeeper"
2. all-caps "sql" -> "SQL" and final period
3. (to separate "SQL" from the commands) rephrase as "using the SQL 
commands UNLOCK TABLE, UNLOCK DATABASE."
4. are the UNLOCK commands part of ZooKeeper? then maybe you should say 
"the ZooKeeper commands" instead of "the SQL commands"

Congrats on fixing some other typos before I saw them!


- Lefty Leverenz


On Oct. 18, 2016, 9:23 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52923/
> ---
> 
> (Updated Oct. 18, 2016, 9:23 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Marta Kuczora, Miklos Csanady, 
> namit jain, Sergio Pena, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-14979
> https://issues.apache.org/jira/browse/HIVE-14979
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding a new configuration option to HiveConf to signal whether stale lock 
> removal is requested on startup.
> Adding a new method to ZooKeeperHiveLockManager to remove stale locks
> Modifying the HiveServer2 to instantiate a lock manager and call the new 
> method if defined by the configuration.
> 
> Please take extra care when reviewing these:
> - Modifying the lock fetching method to use the clientIp from the lock, and 
> not update with the current ip - Not sure why it was done before
> - Instantiation of the lock manager - I might not chose the best method for it
> 
> Open for any suggestions :)
> 
> Thanks,
> Peter
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8ffae3b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
>  14d0ef4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 590b1f3 
> 
> Diff: https://reviews.apache.org/r/52923/diff/
> 
> 
> Testing
> ---
> 
> Created 2 unit test cases:
> - Removing own locks
> - Not removing other server's locks
> 
> Manually tested the Lock manager instantiation method on HiveServer2
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 52923: HIVE-14979 Removing stale Zookeeper locks at HiveServer2 initialization

[Lefty Leverenz]

> On Oct. 18, 2016, 7:28 p.m., Lefty Leverenz wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, lines 1766-1772
> > 
> >
> > 1. camel caps "zookeeper" -> "ZooKeeper"
> > 2. all-caps "sql" -> "SQL" and final period
> > 3. (to separate "SQL" from the commands) rephrase as "using the SQL 
> > commands UNLOCK TABLE, UNLOCK DATABASE."
> > 4. are the UNLOCK commands part of ZooKeeper? then maybe you should say 
> > "the ZooKeeper commands" instead of "the SQL commands"
> > 
> > Congrats on fixing some other typos before I saw them!

Thanks Peter for nudging me to publish my comments.  (Oops.)


- Lefty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52923/#review153057
---


On Oct. 18, 2016, 9:23 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52923/
> ---
> 
> (Updated Oct. 18, 2016, 9:23 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Marta Kuczora, Miklos Csanady, 
> namit jain, Sergio Pena, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-14979
> https://issues.apache.org/jira/browse/HIVE-14979
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding a new configuration option to HiveConf to signal whether stale lock 
> removal is requested on startup.
> Adding a new method to ZooKeeperHiveLockManager to remove stale locks
> Modifying the HiveServer2 to instantiate a lock manager and call the new 
> method if defined by the configuration.
> 
> Please take extra care when reviewing these:
> - Modifying the lock fetching method to use the clientIp from the lock, and 
> not update with the current ip - Not sure why it was done before
> - Instantiation of the lock manager - I might not chose the best method for it
> 
> Open for any suggestions :)
> 
> Thanks,
> Peter
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8ffae3b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/lockmgr/zookeeper/ZooKeeperHiveLockManager.java
>  14d0ef4 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
>   service/src/java/org/apache/hive/service/server/HiveServer2.java 590b1f3 
> 
> Diff: https://reviews.apache.org/r/52923/diff/
> 
> 
> Testing
> ---
> 
> Created 2 unit test cases:
> - Removing own locks
> - Not removing other server's locks
> 
> Manually tested the Lock manager instantiation method on HiveServer2
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 52708: HIVE-14913 Moved system tests to unit tests

[Vineet Garg]

> On Oct. 18, 2016, 6:58 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/acid_globallimit.q, line 16
> > 
> >
> > Worth keeping this second query.

Added this in acid_table_stats


> On Oct. 18, 2016, 6:58 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/orc_ppd_basic.q, lines 216-221
> > 
> >
> > Aren't these tests repeated?

Yes my bad. I missed removing these after getting rid of 
hive.optimize.index.filter conf


> On Oct. 18, 2016, 6:58 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/orc_ppd_basic.q, lines 240-242
> > 
> >
> > Repeated tests.

Thanks for catching these


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52708/#review153119
---


On Oct. 18, 2016, 5:46 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/52708/
> ---
> 
> (Updated Oct. 18, 2016, 5:46 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Bugs: HIVE-14913
> https://issues.apache.org/jira/browse/HIVE-14913
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Moved bunch of system tests to hive unit tests
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 97e310d 
>   ql/src/test/queries/clientpositive/cte_1.q ca4132f 
>   ql/src/test/queries/clientpositive/cte_2.q b49620a 
>   ql/src/test/queries/clientpositive/current_date_timestamp.q 9bed885 
>   ql/src/test/queries/clientpositive/insert1.q 4117b54 
>   ql/src/test/queries/clientpositive/join_acid_non_acid.q 866b466 
>   ql/src/test/queries/clientpositive/lvj_mapjoin.q b726e2a 
>   ql/src/test/queries/clientpositive/orc_ppd_basic.q 43f2c85 
>   ql/src/test/queries/clientpositive/schema_evol_orc_acid_part_update.q 
> d971649 
>   ql/src/test/queries/clientpositive/vectorization_0.q caa6a6a 
>   ql/src/test/results/clientpositive/cte_1.q.out 6e7acf6 
>   ql/src/test/results/clientpositive/cte_2.q.out 4b0bc71 
>   ql/src/test/results/clientpositive/current_date_timestamp.q.out 9c26095 
>   ql/src/test/results/clientpositive/insert1.q.out 7a2c429 
>   ql/src/test/results/clientpositive/llap/cte_1.q.out 41ffdc7 
>   ql/src/test/results/clientpositive/llap/cte_2.q.out 017bded 
>   ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/insert1.q.out 4b30156 
>   ql/src/test/results/clientpositive/llap/join_acid_non_acid.q.out 4905351 
>   ql/src/test/results/clientpositive/llap/lvj_mapjoin.q.out dd4d024 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out aa37ff3 
>   
> ql/src/test/results/clientpositive/llap/schema_evol_orc_acid_part_update.q.out
>  ed698e1 
>   ql/src/test/results/clientpositive/llap/vectorization_0.q.out faceb5c 
>   ql/src/test/results/clientpositive/spark/alter_merge_orc.q.out b5a6d04 
>   ql/src/test/results/clientpositive/spark/vectorization_0.q.out 22fe7cd 
>   ql/src/test/results/clientpositive/vectorization_0.q.out ac33721 
> 
> Diff: https://reviews.apache.org/r/52708/diff/
> 
> 
> Testing
> ---
> 
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

Re: Review Request 52708: HIVE-14913 Moved system tests to unit tests

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52708/
---

(Updated Oct. 18, 2016, 8:37 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-14913
https://issues.apache.org/jira/browse/HIVE-14913


Repository: hive-git


Description
---

Moved bunch of system tests to hive unit tests


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 97e310d 
  ql/src/test/queries/clientpositive/acid_table_stats.q 23d0df4 
  ql/src/test/queries/clientpositive/cte_1.q ca4132f 
  ql/src/test/queries/clientpositive/cte_2.q b49620a 
  ql/src/test/queries/clientpositive/current_date_timestamp.q 9bed885 
  ql/src/test/queries/clientpositive/join_acid_non_acid.q 866b466 
  ql/src/test/queries/clientpositive/lvj_mapjoin.q b726e2a 
  ql/src/test/queries/clientpositive/orc_ppd_basic.q 43f2c85 
  ql/src/test/queries/clientpositive/schema_evol_orc_acid_part_update.q d971649 
  ql/src/test/queries/clientpositive/vectorization_0.q caa6a6a 
  ql/src/test/results/clientpositive/acid_table_stats.q.out d5c509c 
  ql/src/test/results/clientpositive/cte_1.q.out 6e7acf6 
  ql/src/test/results/clientpositive/cte_2.q.out 4b0bc71 
  ql/src/test/results/clientpositive/current_date_timestamp.q.out 9c26095 
  ql/src/test/results/clientpositive/llap/cte_1.q.out 41ffdc7 
  ql/src/test/results/clientpositive/llap/cte_2.q.out 017bded 
  ql/src/test/results/clientpositive/llap/current_date_timestamp.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/llap/join_acid_non_acid.q.out 4905351 
  ql/src/test/results/clientpositive/llap/lvj_mapjoin.q.out dd4d024 
  ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out aa37ff3 
  
ql/src/test/results/clientpositive/llap/schema_evol_orc_acid_part_update.q.out 
ed698e1 
  ql/src/test/results/clientpositive/llap/vectorization_0.q.out faceb5c 
  ql/src/test/results/clientpositive/spark/vectorization_0.q.out 22fe7cd 
  ql/src/test/results/clientpositive/vectorization_0.q.out ac33721 

Diff: https://reviews.apache.org/r/52708/diff/


Testing
---

Pre-commit testing


Thanks,

Vineet Garg

Re: Getting ready for a Hive 1.2.2 release

[Vaibhav Gumashta]

Hi All,

As mentioned by Sushanth earlier, I¹m beginning to work on the 1.2.2
release. I¹m trying to evaluate the stability of the branch by getting a
unit test run. I¹ll post the results shortly with tentative timelines.

Thanks,
‹Vaibhav

On 9/22/16, 10:06 AM, "Sushanth Sowmyan"  wrote:

>Hi Folks,
>
>I'm afraid I've been otherwise occupied and not been able to spend
>enough time on this. Thankfully, Vaibhav Gumashta has volunteered to
>take this on and be the RM for 1.2.2 . He'll follow up on the process
>as it goes forward.
>
>Thanks,
>-Sushanth
>
>
>On Tue, May 3, 2016 at 4:32 PM, Sushanth Sowmyan 
>wrote:
>> Hi All,
>>
>> It has been nearly a year now since 1.2.1 was released, and I said
>> then that I would keep the branch open for further bugfixes with a
>> view of making another 1.2.2 stability upgrade release. There have
>> been 64 such patches committed since then.
>>
>> I think it's time we revisited that and looked to making a 1.2.2
>> release to reflect 1.2.1 + all these updates, and I will go ahead and
>> start rolling out release candidates after next weekend(May 16th)
>> unless anyone has any objections.
>>
>> If anyone wants to get in any other patches before then, please feel
>> free to do so. The original restrictions for commits to branch-1.2,
>> that of no breaking changes, no db changes and no large features still
>> applies, and I'll do a full test verification before pushing it out.
>> If anyone has any patches that they think will take longer than next
>> week, but are important fixes that need to be in this, please ping me.
>> I will edit the wiki (
>> https://cwiki.apache.org/confluence/display/Hive/Hive+1.2+Release+Status
>> ) to reflect this.
>>
>> Thanks,
>> -Sushanth
>

[jira] [Created] (HIVE-15007) Hive 1.2.2 release planning

[Vaibhav Gumashta (JIRA)]

Vaibhav Gumashta created HIVE-15007:
---

 Summary: Hive 1.2.2 release planning
 Key: HIVE-15007
 URL: https://issues.apache.org/jira/browse/HIVE-15007
 Project: Hive
  Issue Type: Task
Affects Versions: 1.2.1
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta


Discussed with [~spena] about triggering unit test runs for 1.2.2 release and 
creating a patch which will trigger precommits looks like a good way.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15008) Cleanup before/after test execution - TestJdbcWithSQLAuthorization, TestJdbcWithSQLAuthUDFBlacklist

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15008:
-

 Summary: Cleanup before/after test execution - 
TestJdbcWithSQLAuthorization, TestJdbcWithSQLAuthUDFBlacklist
 Key: HIVE-15008
 URL: https://issues.apache.org/jira/browse/HIVE-15008
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51755: Support Intersect Distinct

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51755/#review153186
---




ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
(line 956)


Does this need to be public?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
(line 967)


public?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
(line 1000)


need to be public?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
 (line 73)


Add comment about why we only have 2 branches?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
 (line 82)


Nice comments! I think for completeness, also add comments on how NULLs are 
handled. e.g., If they make any difference for all vs distinct.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
 (line 144)


Why can't Gby takes its input from union? If it indeed is necessary, add a 
comment why.



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
 (line 256)


Add a comment on why we need this pass through project?



ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
 (line 285)


determinsitic = true here, isnt it?



ql/src/test/queries/clientpositive/except_all.q (line 56)


Add tests which has aggregates in branches of except so that we can test 
ReduceSinkDedup optimizations with these features.



ql/src/test/queries/clientpositive/except_distinct.q (line 55)


Add tests which has aggregates in branches of except so that we can test 
ReduceSinkDedup optimizations with these features.



ql/src/test/queries/clientpositive/intersect_all.q (line 40)


Add tests which has aggregates in branches of intersect so that we can test 
ReduceSinkDedup optimizations with these features.



ql/src/test/queries/clientpositive/intersect_distinct.q (line 39)


Add tests which has aggregates in branches of intersect so that we can test 
ReduceSinkDedup optimizations with these features.


- Ashutosh Chauhan


On Oct. 15, 2016, 4:10 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51755/
> ---
> 
> (Updated Oct. 15, 2016, 4:10 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12765
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 97e310d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 
> eb0ba7b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> c527e58 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> cf93ed8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExcept.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectMergeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitPullUpConstantsRule.java
>  cc318db 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  8d738aa 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6ab947 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 5d3fa6a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 5c16c55 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 50987c3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBExpr.java cccf0f6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 747f387 
>   
> ql/src/test/org/

Re: Review Request 51755: Support Intersect Distinct

[pengcheng xiong]

> On Oct. 18, 2016, 3:31 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java,
> >  line 41
> > 
> >
> > Any reason for adding Minus as reserved word?

As except is a reserved keyword in SQL2011, although minus is an oracle defined 
operator, i think it makes sense to make it reserved.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51755/#review153098
---


On Oct. 15, 2016, 4:10 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51755/
> ---
> 
> (Updated Oct. 15, 2016, 4:10 a.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12765
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 97e310d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/SimpleFetchOptimizer.java 
> eb0ba7b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
> c527e58 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
> cf93ed8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExcept.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectMergeRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortLimitPullUpConstantsRule.java
>  cc318db 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
>  8d738aa 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6ab947 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 5d3fa6a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 5c16c55 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 50987c3 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBExpr.java cccf0f6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 747f387 
>   
> ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java
>  a427803 
>   ql/src/test/queries/clientpositive/except_all.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/except_distinct.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_all.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_distinct.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/intersect_merge.q PRE-CREATION 
>   ql/src/test/results/clientpositive/auto_join8.q.out ccbafba 
>   ql/src/test/results/clientpositive/except_all.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/except_distinct.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/input25.q.out 465173d 
>   ql/src/test/results/clientpositive/intersect_all.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/intersect_distinct.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/join8.q.out 47b821a 
>   ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 
> 45f2481 
>   ql/src/test/results/clientpositive/smb_mapjoin_25.q.out aba899e 
>   ql/src/test/results/clientpositive/spark/auto_join8.q.out 6909113 
>   ql/src/test/results/clientpositive/spark/join8.q.out 4903c90 
>   ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out eacb438 
>   ql/src/test/results/clientpositive/spark/union_top_level.q.out e1c7fc7 
>   ql/src/test/results/clientpositive/tez/intersect_merge.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/timestamp.q.out 7c08ec8 
>   ql/src/test/results/clientpositive/vector_null_projection.q.out 779f787 
> 
> Diff: https://reviews.apache.org/r/51755/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

[jira] [Created] (HIVE-15009) Additional cleanup from previous test runs when running a batch in ptest

[Siddharth Seth (JIRA)]

Siddharth Seth created HIVE-15009:
-

 Summary: Additional cleanup from previous test runs when running a 
batch in ptest
 Key: HIVE-15009
 URL: https://issues.apache.org/jira/browse/HIVE-15009
 Project: Hive
  Issue Type: Sub-task
Reporter: Siddharth Seth
Assignee: Siddharth Seth






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 51755: Support Intersect Distinct

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51755/
---

(Updated Oct. 19, 2016, 1:36 a.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-12765


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 8aee7f5 
  ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java f308832 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 
c527e58 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelFactories.java 
cf93ed8 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveExcept.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExceptRewriteRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectMergeRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveIntersectRewriteRule.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java
 63aa086 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java d32a0a7 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 5d3fa6a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 5c16c55 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 50987c3 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBExpr.java cccf0f6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 9d58193 
  
ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java
 a427803 
  ql/src/test/queries/clientpositive/except_all.q PRE-CREATION 
  ql/src/test/queries/clientpositive/except_distinct.q PRE-CREATION 
  ql/src/test/queries/clientpositive/intersect_all.q PRE-CREATION 
  ql/src/test/queries/clientpositive/intersect_distinct.q PRE-CREATION 
  ql/src/test/queries/clientpositive/intersect_merge.q PRE-CREATION 
  ql/src/test/results/clientpositive/except_all.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/except_distinct.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/intersect_all.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/intersect_distinct.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/intersect_merge.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/51755/diff/


Testing
---


Thanks,

pengcheng xiong

Re: Start HiveServer2 with Kerberos meet FATAL error

[Micro dong]

[work@tempt48 ~]$ klist -ket /home/work/software/hive/conf/hive.keytab
Keytab name: FILE:/home/work/software/hive/conf/hive.keytab
KVNO Timestamp Principal
 -

   2 10/12/16 15:31:57 hive2/temp...@hadoop.com (aes128-cts-hmac-sha1-96)
   2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des3-cbc-sha1)
   2 10/12/16 15:31:57 hive2/temp...@hadoop.com (arcfour-hmac)
   2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-hmac-sha1)
   2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-cbc-md5)
[work@tempt48 ~]$ hostname
tempt48
[work@tempt48 ~]$ hostname -f
tempt48


2016-10-18 17:48 GMT+08:00 Vivek Shrivastava :

> What is the output of
> klist -ket /home/work/software/hive/conf/hive.keytab
> and
> hostname
> and
> hostname -f
>
> On Tue, Oct 18, 2016 at 4:18 AM, Micro dong 
> wrote:
>
> > tempt48  is the local machine's hostname
> > which is configured in /etc/hosts.
> >
> > 2016-10-18 15:33 GMT+08:00 Chen, Junjie J :
> >
> > > Please try to use full qualified name of host,  such as hive2/
> > > tempt48.hadoop@hadoop.com
> > >
> > > -Original Message-
> > > From: Micro dong [mailto:microle.d...@gmail.com]
> > > Sent: Tuesday, October 18, 2016 2:37 PM
> > > To: dev@hive.apache.org
> > > Subject: Re: Start HiveServer2 with Kerberos meet FATAL error
> > >
> > > I follow your suggestion and it also meet the same error too.
> > >
> > > *start hiveservice*
> > > /home/work/software/hive/bin/hive --service hiveserver2
> > >
> > >
> > > *klist output*
> > > Ticket cache: FILE:/tmp/krb5cc_31100
> > > Default principal: hive2/temp...@hadoop.com
> > >
> > > Valid starting ExpiresService principal
> > > 10/18/16 14:23:04  10/19/16 14:23:04  krbtgt/hadoop@hadoop.com
> renew
> > > until 10/28/16 14:23:04
> > >
> > > 2016-10-18 12:07 GMT+08:00 Vivek Shrivastava  >:
> > >
> > > > can you please change this line
> > > >
> > > > hive2/_h...@hadoop.com * with
> > > > hive2/_h...@hadoop.com*
> > > >
> > > > How are you starting the service, What is the klist output?
> > > >
> > > >
> > > > On Mon, Oct 17, 2016 at 11:09 PM, Micro dong  >
> > > > wrote:
> > > >
> > > > > I'm trying to configure HiveServer2(hive-1.2.1)  With Kerberos。Here
> > > > > is my Hive's configuration file。
> > > > >  **
> > > > > *hive.server2.authentication*
> > > > > *KERBEROS*
> > > > > *  *
> > > > > *  *
> > > > > *hive.server2.authentication.kerberos.principal*
> > > > > *hive2/_h...@hadoop.com *
> > > > > *  *
> > > > > *  *
> > > > > *hive.server2.authentication.kerberos.keytab*
> > > > > */home/work/software/hive/conf/hive.keytab*
> > > > > *  *
> > > > >
> > > > > the keytab file is in its location, its owner is work. But when I
> > > > > try to start the HiveServer2, I see this message on the log:
> > > > >
> > > > > *2016-10-18 10:20:24,867 FATAL [Thread-9]: thrift.ThriftCLIService
> > > > > (ThriftBinaryCLIService.java:run(101)) - Error starting
> HiveServer2:
> > > > could
> > > > > not start ThriftBinaryCLIService*
> > > > > *javax.security.auth.login.LoginException: Kerberos principal
> should
> > > > have
> > > > > 3
> > > > > parts: work**  at *
> > > > > *org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(
> > > > > HiveAuthFactory.java:147)*
> > > > > *atorg.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(
> > > > > ThriftBinaryCLIService.java:58)*
> > > > > *at java.lang.Thread.run(Thread.java:722)*
> > > > >
> > > > > Here *work* is my unix login name. Any help would be highly
> > > appreciated.
> > > > > Best regards
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > 英者自知，雄者自胜。
> > >
> >
> >
> >
> > --
> > 英者自知，雄者自胜。
> >
>



-- 
英者自知，雄者自胜。

Re: Start HiveServer2 with Kerberos meet FATAL error

[Vivek Shrivastava]

This looks good. Please run this one and then try starting the service.

kinit -kt /home/work/software/hive/conf/hive.keytab  hive2/
temp...@hadoop.com


On Tue, Oct 18, 2016 at 10:16 PM, Micro dong  wrote:

> [work@tempt48 ~]$ klist -ket /home/work/software/hive/conf/hive.keytab
> Keytab name: FILE:/home/work/software/hive/conf/hive.keytab
> KVNO Timestamp Principal
>  -
> 
>2 10/12/16 15:31:57 hive2/temp...@hadoop.com (aes128-cts-hmac-sha1-96)
>2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des3-cbc-sha1)
>2 10/12/16 15:31:57 hive2/temp...@hadoop.com (arcfour-hmac)
>2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-hmac-sha1)
>2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-cbc-md5)
> [work@tempt48 ~]$ hostname
> tempt48
> [work@tempt48 ~]$ hostname -f
> tempt48
>
>
> 2016-10-18 17:48 GMT+08:00 Vivek Shrivastava :
>
> > What is the output of
> > klist -ket /home/work/software/hive/conf/hive.keytab
> > and
> > hostname
> > and
> > hostname -f
> >
> > On Tue, Oct 18, 2016 at 4:18 AM, Micro dong 
> > wrote:
> >
> > > tempt48  is the local machine's
> hostname
> > > which is configured in /etc/hosts.
> > >
> > > 2016-10-18 15:33 GMT+08:00 Chen, Junjie J :
> > >
> > > > Please try to use full qualified name of host,  such as hive2/
> > > > tempt48.hadoop@hadoop.com
> > > >
> > > > -Original Message-
> > > > From: Micro dong [mailto:microle.d...@gmail.com]
> > > > Sent: Tuesday, October 18, 2016 2:37 PM
> > > > To: dev@hive.apache.org
> > > > Subject: Re: Start HiveServer2 with Kerberos meet FATAL error
> > > >
> > > > I follow your suggestion and it also meet the same error too.
> > > >
> > > > *start hiveservice*
> > > > /home/work/software/hive/bin/hive --service hiveserver2
> > > >
> > > >
> > > > *klist output*
> > > > Ticket cache: FILE:/tmp/krb5cc_31100
> > > > Default principal: hive2/temp...@hadoop.com
> > > >
> > > > Valid starting ExpiresService principal
> > > > 10/18/16 14:23:04  10/19/16 14:23:04  krbtgt/hadoop@hadoop.com
> > renew
> > > > until 10/28/16 14:23:04
> > > >
> > > > 2016-10-18 12:07 GMT+08:00 Vivek Shrivastava <
> vivshrivast...@gmail.com
> > >:
> > > >
> > > > > can you please change this line
> > > > >
> > > > > hive2/_h...@hadoop.com * with
> > > > > hive2/_h...@hadoop.com*
> > > > >
> > > > > How are you starting the service, What is the klist output?
> > > > >
> > > > >
> > > > > On Mon, Oct 17, 2016 at 11:09 PM, Micro dong <
> microle.d...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > I'm trying to configure HiveServer2(hive-1.2.1)  With
> Kerberos。Here
> > > > > > is my Hive's configuration file。
> > > > > >  **
> > > > > > *hive.server2.authentication*
> > > > > > *KERBEROS*
> > > > > > *  *
> > > > > > *  *
> > > > > > *hive.server2.authentication.kerberos.
> principal*
> > > > > > *hive2/_h...@hadoop.com *
> > > > > > *  *
> > > > > > *  *
> > > > > > *hive.server2.authentication.kerberos.keytab*
> > > > > > */home/work/software/hive/conf/hive.keytab*
> > > > > > *  *
> > > > > >
> > > > > > the keytab file is in its location, its owner is work. But when I
> > > > > > try to start the HiveServer2, I see this message on the log:
> > > > > >
> > > > > > *2016-10-18 10:20:24,867 FATAL [Thread-9]:
> thrift.ThriftCLIService
> > > > > > (ThriftBinaryCLIService.java:run(101)) - Error starting
> > HiveServer2:
> > > > > could
> > > > > > not start ThriftBinaryCLIService*
> > > > > > *javax.security.auth.login.LoginException: Kerberos principal
> > should
> > > > > have
> > > > > > 3
> > > > > > parts: work**  at *
> > > > > > *org.apache.hive.service.auth.HiveAuthFactory.
> getAuthTransFactory(
> > > > > > HiveAuthFactory.java:147)*
> > > > > > *atorg.apache.hive.service.cli.thrift.
> ThriftBinaryCLIService.run(
> > > > > > ThriftBinaryCLIService.java:58)*
> > > > > > *at java.lang.Thread.run(Thread.java:722)*
> > > > > >
> > > > > > Here *work* is my unix login name. Any help would be highly
> > > > appreciated.
> > > > > > Best regards
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > 英者自知，雄者自胜。
> > > >
> > >
> > >
> > >
> > > --
> > > 英者自知，雄者自胜。
> > >
> >
>
>
>
> --
> 英者自知，雄者自胜。
>

Re: Start HiveServer2 with Kerberos meet FATAL error

[Micro dong]

it also meet the same error too.

*2016-10-19 10:28:07,300 FATAL [Thread-9]: thrift.ThriftCLIService
(ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could
not start ThriftBinaryCLIService*
*javax.security.auth.login.LoginException: Kerberos principal should have 3
parts: work*
* at
org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(HiveAuthFactory.java:147)*
* at
org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:58)*
* at java.lang.Thread.run(Thread.java:722)*

2016-10-19 10:22 GMT+08:00 Vivek Shrivastava :

> This looks good. Please run this one and then try starting the service.
>
> kinit -kt /home/work/software/hive/conf/hive.keytab  hive2/
> temp...@hadoop.com
>
>
> On Tue, Oct 18, 2016 at 10:16 PM, Micro dong 
> wrote:
>
> > [work@tempt48 ~]$ klist -ket /home/work/software/hive/conf/hive.keytab
> > Keytab name: FILE:/home/work/software/hive/conf/hive.keytab
> > KVNO Timestamp Principal
> >  -
> > 
> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com
> (aes128-cts-hmac-sha1-96)
> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des3-cbc-sha1)
> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (arcfour-hmac)
> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-hmac-sha1)
> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-cbc-md5)
> > [work@tempt48 ~]$ hostname
> > tempt48
> > [work@tempt48 ~]$ hostname -f
> > tempt48
> >
> >
> > 2016-10-18 17:48 GMT+08:00 Vivek Shrivastava :
> >
> > > What is the output of
> > > klist -ket /home/work/software/hive/conf/hive.keytab
> > > and
> > > hostname
> > > and
> > > hostname -f
> > >
> > > On Tue, Oct 18, 2016 at 4:18 AM, Micro dong 
> > > wrote:
> > >
> > > > tempt48  is the local machine's
> > hostname
> > > > which is configured in /etc/hosts.
> > > >
> > > > 2016-10-18 15:33 GMT+08:00 Chen, Junjie J :
> > > >
> > > > > Please try to use full qualified name of host,  such as hive2/
> > > > > tempt48.hadoop@hadoop.com
> > > > >
> > > > > -Original Message-
> > > > > From: Micro dong [mailto:microle.d...@gmail.com]
> > > > > Sent: Tuesday, October 18, 2016 2:37 PM
> > > > > To: dev@hive.apache.org
> > > > > Subject: Re: Start HiveServer2 with Kerberos meet FATAL error
> > > > >
> > > > > I follow your suggestion and it also meet the same error too.
> > > > >
> > > > > *start hiveservice*
> > > > > /home/work/software/hive/bin/hive --service hiveserver2
> > > > >
> > > > >
> > > > > *klist output*
> > > > > Ticket cache: FILE:/tmp/krb5cc_31100
> > > > > Default principal: hive2/temp...@hadoop.com
> > > > >
> > > > > Valid starting ExpiresService principal
> > > > > 10/18/16 14:23:04  10/19/16 14:23:04  krbtgt/hadoop@hadoop.com
> > > renew
> > > > > until 10/28/16 14:23:04
> > > > >
> > > > > 2016-10-18 12:07 GMT+08:00 Vivek Shrivastava <
> > vivshrivast...@gmail.com
> > > >:
> > > > >
> > > > > > can you please change this line
> > > > > >
> > > > > > hive2/_h...@hadoop.com * with
> > > > > > hive2/_h...@hadoop.com*
> > > > > >
> > > > > > How are you starting the service, What is the klist output?
> > > > > >
> > > > > >
> > > > > > On Mon, Oct 17, 2016 at 11:09 PM, Micro dong <
> > microle.d...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I'm trying to configure HiveServer2(hive-1.2.1)  With
> > Kerberos。Here
> > > > > > > is my Hive's configuration file。
> > > > > > >  **
> > > > > > > *hive.server2.authentication*
> > > > > > > *KERBEROS*
> > > > > > > *  *
> > > > > > > *  *
> > > > > > > *hive.server2.authentication.kerberos.
> > principal*
> > > > > > > *hive2/_h...@hadoop.com *
> > > > > > > *  *
> > > > > > > *  *
> > > > > > > *hive.server2.authentication.kerberos.keytab*
> > > > > > > */home/work/software/hive/conf/hive.keytab*
> > > > > > > *  *
> > > > > > >
> > > > > > > the keytab file is in its location, its owner is work. But
> when I
> > > > > > > try to start the HiveServer2, I see this message on the log:
> > > > > > >
> > > > > > > *2016-10-18 10:20:24,867 FATAL [Thread-9]:
> > thrift.ThriftCLIService
> > > > > > > (ThriftBinaryCLIService.java:run(101)) - Error starting
> > > HiveServer2:
> > > > > > could
> > > > > > > not start ThriftBinaryCLIService*
> > > > > > > *javax.security.auth.login.LoginException: Kerberos principal
> > > should
> > > > > > have
> > > > > > > 3
> > > > > > > parts: work**  at *
> > > > > > > *org.apache.hive.service.auth.HiveAuthFactory.
> > getAuthTransFactory(
> > > > > > > HiveAuthFactory.java:147)*
> > > > > > > *atorg.apache.hive.service.cli.thrift.
> > ThriftBinaryCLIService.run(
> > > > > > > ThriftBinaryCLIService.java:58)*
> > > > > > > *at java.lang.Thread.run(Thread.java:722)*
> > > > > > >
> > > > > > > Here *work* is my unix login name. Any help would be highly
> > > > > appreciated.
> > > > > > > Best regards
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >

Re: Start HiveServer2 with Kerberos meet FATAL error

[Micro dong]

It seems  to me it's trying to login into HiveServer with unix username
instead of  Kerberos keytab

2016-10-19 10:30 GMT+08:00 Micro dong :

> it also meet the same error too.
>
> *2016-10-19 10:28:07,300 FATAL [Thread-9]: thrift.ThriftCLIService
> (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2: could
> not start ThriftBinaryCLIService*
> *javax.security.auth.login.LoginException: Kerberos principal should have
> 3 parts: work*
> * at
> org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(HiveAuthFactory.java:147)*
> * at
> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:58)*
> * at java.lang.Thread.run(Thread.java:722)*
>
> 2016-10-19 10:22 GMT+08:00 Vivek Shrivastava :
>
>> This looks good. Please run this one and then try starting the service.
>>
>> kinit -kt /home/work/software/hive/conf/hive.keytab  hive2/
>> temp...@hadoop.com
>>
>>
>> On Tue, Oct 18, 2016 at 10:16 PM, Micro dong 
>> wrote:
>>
>> > [work@tempt48 ~]$ klist -ket /home/work/software/hive/conf/hive.keytab
>> > Keytab name: FILE:/home/work/software/hive/conf/hive.keytab
>> > KVNO Timestamp Principal
>> >  -
>> > 
>> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com
>> (aes128-cts-hmac-sha1-96)
>> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des3-cbc-sha1)
>> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (arcfour-hmac)
>> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-hmac-sha1)
>> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-cbc-md5)
>> > [work@tempt48 ~]$ hostname
>> > tempt48
>> > [work@tempt48 ~]$ hostname -f
>> > tempt48
>> >
>> >
>> > 2016-10-18 17:48 GMT+08:00 Vivek Shrivastava > >:
>> >
>> > > What is the output of
>> > > klist -ket /home/work/software/hive/conf/hive.keytab
>> > > and
>> > > hostname
>> > > and
>> > > hostname -f
>> > >
>> > > On Tue, Oct 18, 2016 at 4:18 AM, Micro dong 
>> > > wrote:
>> > >
>> > > > tempt48  is the local machine's
>> > hostname
>> > > > which is configured in /etc/hosts.
>> > > >
>> > > > 2016-10-18 15:33 GMT+08:00 Chen, Junjie J > >:
>> > > >
>> > > > > Please try to use full qualified name of host,  such as hive2/
>> > > > > tempt48.hadoop@hadoop.com
>> > > > >
>> > > > > -Original Message-
>> > > > > From: Micro dong [mailto:microle.d...@gmail.com]
>> > > > > Sent: Tuesday, October 18, 2016 2:37 PM
>> > > > > To: dev@hive.apache.org
>> > > > > Subject: Re: Start HiveServer2 with Kerberos meet FATAL error
>> > > > >
>> > > > > I follow your suggestion and it also meet the same error too.
>> > > > >
>> > > > > *start hiveservice*
>> > > > > /home/work/software/hive/bin/hive --service hiveserver2
>> > > > >
>> > > > >
>> > > > > *klist output*
>> > > > > Ticket cache: FILE:/tmp/krb5cc_31100
>> > > > > Default principal: hive2/temp...@hadoop.com
>> > > > >
>> > > > > Valid starting ExpiresService principal
>> > > > > 10/18/16 14:23:04  10/19/16 14:23:04  krbtgt/
>> hadoop@hadoop.com
>> > > renew
>> > > > > until 10/28/16 14:23:04
>> > > > >
>> > > > > 2016-10-18 12:07 GMT+08:00 Vivek Shrivastava <
>> > vivshrivast...@gmail.com
>> > > >:
>> > > > >
>> > > > > > can you please change this line
>> > > > > >
>> > > > > > hive2/_h...@hadoop.com * with
>> > > > > > hive2/_h...@hadoop.com*
>> > > > > >
>> > > > > > How are you starting the service, What is the klist output?
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Oct 17, 2016 at 11:09 PM, Micro dong <
>> > microle.d...@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > I'm trying to configure HiveServer2(hive-1.2.1)  With
>> > Kerberos。Here
>> > > > > > > is my Hive's configuration file。
>> > > > > > >  **
>> > > > > > > *hive.server2.authentication*
>> > > > > > > *KERBEROS*
>> > > > > > > *  *
>> > > > > > > *  *
>> > > > > > > *hive.server2.authentication.kerberos.
>> > principal*
>> > > > > > > *hive2/_h...@hadoop.com *
>> > > > > > > *  *
>> > > > > > > *  *
>> > > > > > > *hive.server2.authenticat
>> ion.kerberos.keytab*
>> > > > > > > */home/work/software/hiv
>> e/conf/hive.keytab*
>> > > > > > > *  *
>> > > > > > >
>> > > > > > > the keytab file is in its location, its owner is work. But
>> when I
>> > > > > > > try to start the HiveServer2, I see this message on the log:
>> > > > > > >
>> > > > > > > *2016-10-18 10:20:24,867 FATAL [Thread-9]:
>> > thrift.ThriftCLIService
>> > > > > > > (ThriftBinaryCLIService.java:run(101)) - Error starting
>> > > HiveServer2:
>> > > > > > could
>> > > > > > > not start ThriftBinaryCLIService*
>> > > > > > > *javax.security.auth.login.LoginException: Kerberos principal
>> > > should
>> > > > > > have
>> > > > > > > 3
>> > > > > > > parts: work**  at *
>> > > > > > > *org.apache.hive.service.auth.HiveAuthFactory.
>> > getAuthTransFactory(
>> > > > > > > HiveAuthFactory.java:147)*
>> > > > > > > *atorg.apache.hive.service.cli.thrift.
>> > ThriftBinaryCL

[jira] [Created] (HIVE-15010) Make LockComponent aware if it's part of dynamic partition operation

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-15010:
-

 Summary: Make LockComponent aware if it's part of dynamic 
partition operation
 Key: HIVE-15010
 URL: https://issues.apache.org/jira/browse/HIVE-15010
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Eugene Koifman
Assignee: Eugene Koifman


This is needed by HIVE-10924 but I want to make sure it's a separate patch to 
make any back porting easier



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15011) fix issues with MoveTask.releaseLocks()

[Eugene Koifman (JIRA)]

Eugene Koifman created HIVE-15011:
-

 Summary: fix issues with MoveTask.releaseLocks()
 Key: HIVE-15011
 URL: https://issues.apache.org/jira/browse/HIVE-15011
 Project: Hive
  Issue Type: Sub-task
Reporter: Eugene Koifman
Assignee: Eugene Koifman


in Merge we can have multiple move tasks so releasing the locks should be done 
on from the "last one" - in practice they run concurrently

see if there is a quick fix for short term

(slightly) longer term - get rid of MoveTask for Acid writes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Start HiveServer2 with Kerberos meet FATAL error

[Vivek Shrivastava]

How are you starting the service? Do you have Hadoop setup done and
integrated with Kerberos?

On Tue, Oct 18, 2016 at 10:33 PM, Micro dong  wrote:

> It seems  to me it's trying to login into HiveServer with unix username
> instead of  Kerberos keytab
>
> 2016-10-19 10:30 GMT+08:00 Micro dong :
>
> > it also meet the same error too.
> >
> > *2016-10-19 10:28:07,300 FATAL [Thread-9]: thrift.ThriftCLIService
> > (ThriftBinaryCLIService.java:run(101)) - Error starting HiveServer2:
> could
> > not start ThriftBinaryCLIService*
> > *javax.security.auth.login.LoginException: Kerberos principal should
> have
> > 3 parts: work*
> > * at
> > org.apache.hive.service.auth.HiveAuthFactory.getAuthTransFactory(
> HiveAuthFactory.java:147)*
> > * at
> > org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.
> run(ThriftBinaryCLIService.java:58)*
> > * at java.lang.Thread.run(Thread.java:722)*
> >
> > 2016-10-19 10:22 GMT+08:00 Vivek Shrivastava :
> >
> >> This looks good. Please run this one and then try starting the service.
> >>
> >> kinit -kt /home/work/software/hive/conf/hive.keytab  hive2/
> >> temp...@hadoop.com
> >>
> >>
> >> On Tue, Oct 18, 2016 at 10:16 PM, Micro dong 
> >> wrote:
> >>
> >> > [work@tempt48 ~]$ klist -ket /home/work/software/hive/conf/
> hive.keytab
> >> > Keytab name: FILE:/home/work/software/hive/conf/hive.keytab
> >> > KVNO Timestamp Principal
> >> >  -
> >> > 
> >> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com
> >> (aes128-cts-hmac-sha1-96)
> >> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des3-cbc-sha1)
> >> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (arcfour-hmac)
> >> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-hmac-sha1)
> >> >2 10/12/16 15:31:57 hive2/temp...@hadoop.com (des-cbc-md5)
> >> > [work@tempt48 ~]$ hostname
> >> > tempt48
> >> > [work@tempt48 ~]$ hostname -f
> >> > tempt48
> >> >
> >> >
> >> > 2016-10-18 17:48 GMT+08:00 Vivek Shrivastava <
> vivshrivast...@gmail.com
> >> >:
> >> >
> >> > > What is the output of
> >> > > klist -ket /home/work/software/hive/conf/hive.keytab
> >> > > and
> >> > > hostname
> >> > > and
> >> > > hostname -f
> >> > >
> >> > > On Tue, Oct 18, 2016 at 4:18 AM, Micro dong  >
> >> > > wrote:
> >> > >
> >> > > > tempt48  is the local machine's
> >> > hostname
> >> > > > which is configured in /etc/hosts.
> >> > > >
> >> > > > 2016-10-18 15:33 GMT+08:00 Chen, Junjie J <
> junjie.j.c...@intel.com
> >> >:
> >> > > >
> >> > > > > Please try to use full qualified name of host,  such as hive2/
> >> > > > > tempt48.hadoop@hadoop.com
> >> > > > >
> >> > > > > -Original Message-
> >> > > > > From: Micro dong [mailto:microle.d...@gmail.com]
> >> > > > > Sent: Tuesday, October 18, 2016 2:37 PM
> >> > > > > To: dev@hive.apache.org
> >> > > > > Subject: Re: Start HiveServer2 with Kerberos meet FATAL error
> >> > > > >
> >> > > > > I follow your suggestion and it also meet the same error too.
> >> > > > >
> >> > > > > *start hiveservice*
> >> > > > > /home/work/software/hive/bin/hive --service hiveserver2
> >> > > > >
> >> > > > >
> >> > > > > *klist output*
> >> > > > > Ticket cache: FILE:/tmp/krb5cc_31100
> >> > > > > Default principal: hive2/temp...@hadoop.com
> >> > > > >
> >> > > > > Valid starting ExpiresService principal
> >> > > > > 10/18/16 14:23:04  10/19/16 14:23:04  krbtgt/
> >> hadoop@hadoop.com
> >> > > renew
> >> > > > > until 10/28/16 14:23:04
> >> > > > >
> >> > > > > 2016-10-18 12:07 GMT+08:00 Vivek Shrivastava <
> >> > vivshrivast...@gmail.com
> >> > > >:
> >> > > > >
> >> > > > > > can you please change this line
> >> > > > > >
> >> > > > > > hive2/_h...@hadoop.com * with
> >> > > > > > hive2/_h...@hadoop.com*
> >> > > > > >
> >> > > > > > How are you starting the service, What is the klist output?
> >> > > > > >
> >> > > > > >
> >> > > > > > On Mon, Oct 17, 2016 at 11:09 PM, Micro dong <
> >> > microle.d...@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > I'm trying to configure HiveServer2(hive-1.2.1)  With
> >> > Kerberos。Here
> >> > > > > > > is my Hive's configuration file。
> >> > > > > > >  **
> >> > > > > > > *hive.server2.authentication*
> >> > > > > > > *KERBEROS*
> >> > > > > > > *  *
> >> > > > > > > *  *
> >> > > > > > > *hive.server2.authentication.kerberos.
> >> > principal*
> >> > > > > > > *hive2/_h...@hadoop.com  >*
> >> > > > > > > *  *
> >> > > > > > > *  *
> >> > > > > > > *hive.server2.authenticat
> >> ion.kerberos.keytab*
> >> > > > > > > */home/work/software/hiv
> >> e/conf/hive.keytab*
> >> > > > > > > *  *
> >> > > > > > >
> >> > > > > > > the keytab file is in its location, its owner is work. But
> >> when I
> >> > > > > > > try to start the HiveServer2, I see this message on the log:
> >> > > > > > >
> >> > > > > > > *2016-10-18 10:20:24,867 FATAL [Thread-9]:
> >> > thrift.ThriftCLIService
> >> > > > > > > (ThriftBinaryCLIS

[jira] [Created] (HIVE-15012) array_contains type error

[john zhao (JIRA)]

john zhao created HIVE-15012:


 Summary: array_contains type error
 Key: HIVE-15012
 URL: https://issues.apache.org/jira/browse/HIVE-15012
 Project: Hive
  Issue Type: Bug
 Environment: Hive 2.1.0
Subversion 
git://jcamachguezrMBP/Users/jcamachorodriguez/src/workspaces/hive/HIVE-release2/hive
 -r 9265bc24d75ac945bde9ce1a0999fddd8f2aae29
Compiled by jcamachorodriguez on Fri Jun 17 01:03:25 BST 2016
>From source with checksum 1f896b8fae57fbd29b047d6d67b75f3c
Reporter: john zhao


When I use array_contains in where like this: WHERE 
array_contains(array(38461L), aduserid)
I got an error: FAILED: SemanticException [Error 10016]: Line 3:35 Argument 
type mismatch 'aduserid': "int" expected at function ARRAY_CONTAINS, but 
"bigint" is found



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)