Re: Review Request 58181: HIVE-15986 Support for 'is [NOT] distinct from'

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58181/
---

(Updated April 6, 2017, 5:38 a.m.)


Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.


Bugs: HIVE-15986
https://issues.apache.org/jira/browse/HIVE-15986


Repository: hive-git


Description
---

This patch adds support for 'is distinct from' and 'is not distinct from'.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 7a70c9c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ccfb455 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
 85450c9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8c4ee8a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java f979c14 
  ql/src/test/queries/clientpositive/is_distinct_from.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/is_distinct_from.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 68e248a 
  ql/src/test/results/clientpositive/udf_equal.q.out 52bd843 


Diff: https://reviews.apache.org/r/58181/diff/4/

Changes: https://reviews.apache.org/r/58181/diff/3-4/


Testing
---

Added new tests
Pre-commit testing


Thanks,

Vineet Garg

[jira] [Created] (HIVE-16396) Sync storage-api version in pom.xml

[Teddy Choi (JIRA)]

Teddy Choi created HIVE-16396:
-

 Summary: Sync storage-api version in pom.xml
 Key: HIVE-16396
 URL: https://issues.apache.org/jira/browse/HIVE-16396
 Project: Hive
  Issue Type: Bug
  Components: storage-api
Reporter: Teddy Choi
Assignee: Teddy Choi


Apache Hive 3.0 still requires storage-api.version of 2.3.0-SNAPSHOT. It causes 
some bugs when storage-api module has some changes. It needs to be synced with 
Hive project version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16395) ConcurrentModificationException on config object in HoS

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-16395:
---

 Summary: ConcurrentModificationException on config object in HoS
 Key: HIVE-16395
 URL: https://issues.apache.org/jira/browse/HIVE-16395
 Project: Hive
  Issue Type: Task
  Components: Spark
Reporter: Sahil Takiar


Looks like this is happening inside spark executors, looks to be some race 
condition when modifying {{Configuration}} objects.

Stack-Trace:

{quote}
java.io.IOException: java.lang.reflect.InvocationTargetException
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:267)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.(HadoopShimsSecure.java:213)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:334)
at 
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:682)
at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:240)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at 
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:242)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at 
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:253)
... 21 more
Caused by: java.util.ConcurrentModificationException
at java.util.Hashtable$Enumerator.next(Hashtable.java:1167)
at 
org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2455)
at 
org.apache.hadoop.fs.s3a.S3AUtils.propagateBucketOptions(S3AUtils.java:716)
at 
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:181)
at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2815)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2852)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2834)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at 
org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:108)
at 
org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at 
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.(CombineHiveRecordReader.java:68)
... 26 more
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] split metastore and service

[Carl Steinbach]

+1!

On Tue, Mar 28, 2017 at 6:36 PM, Thejas Nair  wrote:

> Also, thanks for the email thread to bring peoples attention to this
> change.
>
> On Tue, Mar 28, 2017 at 6:35 PM, Thejas Nair 
> wrote:
>
> > +1
> > Thanks for looking into this!
> >
> >
> > On Tue, Mar 28, 2017 at 11:26 AM, Eugene Koifman <
> ekoif...@hortonworks.com
> > > wrote:
> >
> >> +1 reduce the number of uber jars
> >>
> >>
> >> On 3/27/17, 1:05 PM, "Sergey Shelukhin"  wrote:
> >>
> >> Splitting the metastore would also allow us to get rid of compile
> time
> >> dependencies that are resolved via reflection right now.
> >> +1 on the feature
> >>
> >> On 17/3/27, 07:33, "Zoltan Haindrich" 
> >> wrote:
> >>
> >> >Hello,
> >> >
> >> >Currently the jdbc driver contains lots of hive code; which are not
> >> >needed for the driver to function properly - jdbc-standalone is
> >> currently
> >> >a 60M binary! :)
> >> >
> >> >I've opened a ticket, to explore the possibilites what can be done
> in
> >> >this aspect to reduce jdbc's dependencies.
> >> >
> >> >I was able to remove most of the service and the metastore
> >> dependencies -
> >> >by introducing 2 new modules: I called them metastore-api and
> >> >service-client.
> >> >As a change like this would mean that the released jars name and
> >> purpose
> >> >would change - I didn't wanted to just file a jira about it :)
> >> >
> >> >So...I would like to ask for opinions or any concerns against doing
> >> the
> >> >following:
> >> >
> >> >1) Splitting the metastore module; the new module would be named as
> >> >metastore-X (my proposals for X are: client,rpc,if or api).
> >> >  * the dependency would contain the thrift interface
> >> >  * and possibly a few other source files which are needed to use
> it.
> >> >
> >> >2) Splitting the service module; the new module would be named
> >> service-X
> >> >(my propsal for X would be client)
> >> >  * the module would contain auth related classes
> >> >  * some other basic stuffs like RowSet
> >> >  * connected change: jdbc driver would change the support of
> >> embedded
> >> >mode to only make it usable if 'service' is loaded onto the
> classpath
> >> >
> >> >With these two modules available, the size of the jdbc driver have
> >> >dropped to about 21M.
> >> >
> >> >more info:
> >> >https://issues.apache.org/jira/browse/HIVE-16214
> >> >
> >> >regards,
> >> >Zoltan
> >>
> >>
> >>
> >>
> >
>

[jira] [Created] (HIVE-16394) HoS does not support queue name change in middle of session

[Chaoyu Tang (JIRA)]

Chaoyu Tang created HIVE-16394:
--

 Summary: HoS does not support queue name change in middle of 
session
 Key: HIVE-16394
 URL: https://issues.apache.org/jira/browse/HIVE-16394
 Project: Hive
  Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang


The mapreduce.job.queuename only effects when HoS executes its query first 
time. After that, changing mapreduce.job.queuename won't change the query yarn 
scheduler queue name.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Apache Hive metastore and Impala

[Jim Apple]

On Wed, Apr 5, 2017 at 4:41 PM, Edward Capriolo 
wrote:

> Hello impala devs!
>
> Let me say that I have used impala a lot and am very impressed with it.
>

Thank you!


> I am not asserting that impala if affected by case #1 or #2 currently, but
> something to be aware of. If you need guidance  feel free to discuss
> further with the Hive PMC.
>

I don't think there is a danger right now of Apache Impala (incubating)
forking Hive. As far as artifact publication, Apache Impala (incubating)
does not publish binary artifacts at this time. Nonetheless, I will forward
your message to some people at Cloudera who might be interested, since
Cloudera does publish binaries.


> Apache Hive proposes and adds syntax all the time. For example, this
> feature is in the works now (
> https://issues.apache.org/jira/browse/HIVE-15986).


Funny you should mention that feature -- I actually implemented that for
Impala:
https://github.com/apache/incubator-impala/commit/1a3d7ffd4fd392b3ed831dfc7a3bfcfdb8cb8bbd#diff-a7c8505823aef79d508ede7e4d4e464a

As you say, though, our syntax is unlikely to be identical. I've sent a
patch for review that will change our webpage:
http://gerrit.cloudera.org:8080/6567

Review Request 58227: Hive-16393: Fix visibility of CodahaleReporter interface

[Sunitha Beeram via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58227/
---

Review request for hive, Carl Steinbach and Ratandeep Ratti.


Bugs: HIVE-16393
https://issues.apache.org/jira/browse/HIVE-16393


Repository: hive-git


Description
---

Hive-16393: Fix visibility of CodahaleReporter interface


Diffs
-

  
common/src/java/org/apache/hadoop/hive/common/metrics/metrics2/CodahaleReporter.java
 ba4d8e47c60df64bd935716f49c0b07864d48860 


Diff: https://reviews.apache.org/r/58227/diff/1/


Testing
---

None.


Thanks,

Sunitha Beeram

[jira] [Created] (HIVE-16393) Fix visibility of CodahaleReporter interface

[Sunitha Beeram (JIRA)]

Sunitha Beeram created HIVE-16393:
-

 Summary: Fix visibility of CodahaleReporter interface
 Key: HIVE-16393
 URL: https://issues.apache.org/jira/browse/HIVE-16393
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Sunitha Beeram
Assignee: Sunitha Beeram


CodahaleReporter interface, introduced via Hive-16206 has package-private 
visibility. This prevents external libraries from extending it.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Apache Hive metastore and Impala

[Edward Capriolo]

Hello impala devs!

Let me say that I have used impala a lot and am very impressed with it.

I know impala is moving into the Apache incubator (I have an incubator
prodling gossip so I know this is challenging). There are few things I want
to bring to your attention/discuss, so that they do not become an issue or
blocker in the future.

1) code
Your proposal https://wiki.apache.org/incubator/ImpalaProposal lists hive
as a dependency.

External Dependencies

Apache Hive (Apache Software License v2.0)

I notice that the cloudera impala has CDH "hive" (which are rather old)
jars in its source tree:

https://github.com/cloudera/Impala/tree/8b621a301329d91fbe10a8aac5e39a2b14d6d25f/thirdparty/hive-1.1.0-cdh5.12.0-SNAPSHOT

A quick search did not find any evidence of that in incubator-impala (which
is good):
https://github.com/apache/incubator-impala/

We (Hive) want people using only official Apache Hive releases for
dependencies. We want to avoid:
1) Full or partial code forks of Apache Hive which still carry the Hive name
2) Artifacts published to central repositories named "*Hive*" which could
be confusing

I am not asserting that impala if affected by case #1 or #2 currently, but
something to be aware of. If you need guidance  feel free to discuss
further with the Hive PMC.

2) Next topic, the Hive name and statements that imply compatibility:

http://impala.apache.org/

For Apache Hive users, Impala utilizes the same metadata, ODBC driver, SQL
syntax, and user interface as Hive—so you don't have to worry about
re-inventing the implementation wheel.

Apache Hive proposes and adds syntax all the time. For example, this
feature is in the works now (
https://issues.apache.org/jira/browse/HIVE-15986). Even if every effort was
made to keep the languages and features in sync no one would be able to
make this claim. This because Apache Hive does not have compatibility tests
for any of these things (We do not have anything like ANSI SQL 92).

This text needs be replaced. It is probably fine to make statements such as
"Impala can run many of queries as Apache Hive", or "users of Apache Hive
will find many familiar features in Impala".

Again welcome to the incubator, I am sure getting impala through is fun
with the c++ ness of it all!

Thanks,
Edward

Re: [VOTE] Apache Hive 1.2.2 Release Candidate 0

[Sergio Pena]

+1 (no-binding)

I unpacked the bin and src packages.
Verified gpg and md5 signatures.
Check license and release notes files.
Run a few queries from hive-cli.

- Sergio

On Tue, Apr 4, 2017 at 11:12 AM, Ashutosh Chauhan 
wrote:

> Verified md5 of src and binary tar balls.
> Built from src.
> Ran some simple queries like join, group by.
> All looks good.
>
> +1
>
> Thanks,
> Ashutosh
>
> On Mon, Apr 3, 2017 at 4:47 PM, Vaibhav Gumashta <
> vgumas...@hortonworks.com>
> wrote:
>
> > Thanks for pointing out Ashutosh. Link to my PGP key:
> > http://pgp.mit.edu/pks/lookup?search=gumashta=index.
> >
> > I think it will take a day or so for the KEYS file to be updated (it is
> > auto generated), but if you want to test the release in the meantime,
> > please use the above link to access the signing key.
> >
> > Thanks,
> > ‹Vaibhav
> >
> > On 4/3/17, 2:53 PM, "Ashutosh Chauhan"  wrote:
> >
> > >Hi Vaibhav,
> > >
> > >Can't locate your key at any of standard location. Can you point out
> which
> > >key you used to sign the release?
> > >
> > >Thanks,
> > >Ashutosh
> > >
> > >On Mon, Apr 3, 2017 at 12:51 AM, Vaibhav Gumashta
> > > > >> wrote:
> > >> Hi everyone,
> > >>
> > >> Apache Hive 1.2.2 Release Candidate 0 is available here:
> > >>
> > >> https://dist.apache.org/repos/dist/dev/hive/apache-hive-1.2.2-rc0/
> > >>
> > >> Maven artifacts are available here:
> > >>
> > >> https://repository.apache.org/content/repositories/
> orgapachehive-1072/
> > >>
> > >> Source tag for RC0 is at:
> > >> https://github.com/apache/hive/releases/tag/release-1.2.2-rc0
> > >>
> > >> Voting will conclude in 72 hours.
> > >>
> > >> Hive PMC Members: Please test and vote.
> > >>
> > >> Thanks,
> > >> -Vaibhav
> > >>
> > >>
> >
> >
>

[jira] [Created] (HIVE-16392) Remove hive.warehouse.subdir.inherit.perms and all permissions inheritance logic

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-16392:
---

 Summary: Remove hive.warehouse.subdir.inherit.perms and all 
permissions inheritance logic
 Key: HIVE-16392
 URL: https://issues.apache.org/jira/browse/HIVE-16392
 Project: Hive
  Issue Type: Task
  Components: Security
Reporter: Sahil Takiar
Assignee: Sahil Takiar


As discussed in HIVE-16346 we should remove the config 
{{hive.warehouse.subdir.inherit.perms}} and all the permissions inheritance 
logic.

This feature is no longer needed in Hive as the traditional permission model 
has largely been replaced by external security systems such as Ranger and 
Sentry.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

[Reynold Xin (JIRA)]

Reynold Xin created HIVE-16391:
--

 Summary: Publish proper Hive 1.2 jars (without including all 
dependencies in uber jar)
 Key: HIVE-16391
 URL: https://issues.apache.org/jira/browse/HIVE-16391
 Project: Hive
  Issue Type: Task
  Components: Build Infrastructure
Reporter: Reynold Xin


Apache Spark currently depends on a forked version of Apache Hive. AFAIK, the 
only change in the fork is to work around the issue that Hive publishes only 
two sets of jars: one set with no dependency declared, and another with all the 
dependencies included in the published uber jar.

There is general consensus on both sides that we should remove the forked Hive.

The change in the forked version is recorded here 
https://github.com/JoshRosen/hive/tree/release-1.2.1-spark2


Note that the fork in the past included other fixes but those have all become 
unnecessary.





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58181: HIVE-15986 Support for 'is [NOT] distinct from'

[Vineet Garg]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58181/
---

(Updated April 5, 2017, 9:53 p.m.)


Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.


Changes
---

Test fix


Bugs: HIVE-15986
https://issues.apache.org/jira/browse/HIVE-15986


Repository: hive-git


Description
---

This patch adds support for 'is distinct from' and 'is not distinct from'.


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 7a70c9c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ccfb455 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
 85450c9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8c4ee8a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java f979c14 
  ql/src/test/queries/clientpositive/is_distinct_from.q PRE-CREATION 
  ql/src/test/results/clientpositive/llap/is_distinct_from.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 68e248a 
  ql/src/test/results/clientpositive/udf_equal.q.out 52bd843 


Diff: https://reviews.apache.org/r/58181/diff/3/

Changes: https://reviews.apache.org/r/58181/diff/2-3/


Testing
---

Added new tests
Pre-commit testing


Thanks,

Vineet Garg

Re: Review Request 58203: HIVE-16345 BeeLineDriver should be able to run qtest files which are using default database tables

[Yongzhi Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58203/#review171166
---




itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java
Lines 130 (patched)


How do you handle the case command has comment following ';' and new 
command start after ;  ? Do these cases matters?
For example:
show tables; --comment

show tables; select * from
src;

The beeline.Commands class has code similar to getCommands:
handleMultiLineCmd, logic in execute
Could you figure out a way to use the some of the code there?



itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java
Lines 160 (patched)


Is that possible the table belong to other database?
For example:
use foo;
select * from tableinfoo;



itests/util/src/main/java/org/apache/hive/beeline/qfile/QFileBeeLineClient.java
Line 92 (original), 90 (patched)


Why we need to replace the tablename with default.tablename? Could you just 
add use default ?


- Yongzhi Chen


On April 5, 2017, 10:35 a.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58203/
> ---
> 
> (Updated April 5, 2017, 10:35 a.m.)
> 
> 
> Review request for hive, Aihua Xu, Zoltan Haindrich, Yongzhi Chen, and Barna 
> Zsombor Klara.
> 
> 
> Bugs: HIVE-16345
> https://issues.apache.org/jira/browse/HIVE-16345
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The goal of the change is to run qtest files which contain queries on tables 
> created by the init scripts.
> It adds the possibility to rewrite the src table references to default.src
> 
> This patch contains the following changes:
> - Added new parameter to the driver, to control weather the rewrite the table 
> names or not (test.rewrite.source.tables) - default is true
> - Made QTestUtil.getSrcTables() available for QFile class
> - Run the QFile not with "!run testfile.q", but reading the file, and 
> assembling the commands - enable us to parse the queries, and provide better 
> feedback about the failing queries
> - QFile rewrites the source tables, if it is required
> - Used 9 qtest files from the CliDriver, and added them to BeeLine tests
> - Added new filters, and removed redundant ones - I was able to remove every 
> QFile specific filter, and corresponding setter methods as well
> - Moved QFile classes to org.apache.hive.beeline package, so it can use 
> package private methods from BeeLine, and Commands
> - Refactored needsContinuation method in BeeLine, so it can be called from a 
> static context as well
> 
> And one important change is:
> - In Utilities.setMapRedWork, change the INPUT_NAME value in the conf to a 
> mapreduce task specific value. This one is used by the IOContextMap to cache 
> the IOContext objects. Using the same value for every mapred task prevented 
> them to run in the same JVM. The test were running sequencially, but failed 
> randomly in parallel
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLine.java 11526a7 
>   itests/src/test/resources/testconfiguration.properties 7a70c9c 
>   
> itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
>  0d63f5d 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2abf252 
>   itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java ae5a349 
>   
> itests/util/src/main/java/org/apache/hive/beeline/qfile/QFileBeeLineClient.java
>  760fde6 
>   itests/util/src/main/java/org/apache/hive/beeline/qfile/package-info.java 
> fcd50ec 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 79955e9 
>   ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 
> 385f9b7 
>   ql/src/test/results/clientpositive/beeline/escape_comments.q.out abc0fee 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_11.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_12.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_13.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_16.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_3.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out PRE-CREATION 
> 
> 
> Diff: 

[jira] [Created] (HIVE-16390) LLAP IO should take job config into account; also LLAP config should load defaults

[Sergey Shelukhin (JIRA)]

Sergey Shelukhin created HIVE-16390:
---

 Summary: LLAP IO should take job config into account; also LLAP 
config should load defaults
 Key: HIVE-16390
 URL: https://issues.apache.org/jira/browse/HIVE-16390
 Project: Hive
  Issue Type: Bug
Reporter: Siddharth Seth
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16389) Allow HookContext to access SQLOperationDisplay

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-16389:
---

 Summary: Allow HookContext to access SQLOperationDisplay
 Key: HIVE-16389
 URL: https://issues.apache.org/jira/browse/HIVE-16389
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Sahil Takiar
Assignee: Sahil Takiar


There is a lot of useful information in {{SQLOperationDisplay}} that users of 
Hive Hooks may be interested in.

We should allow Hive Hooks to access this info by adding the 
{{SQLOperationDisplay}} to {{HookContext}}.

This will allow hooks to have access to all information available in the HS2 
Web UI.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58181: HIVE-15986 Support for 'is [NOT] distinct from'

[Vineet Garg]

> On April 5, 2017, 3:37 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
> > Lines 590-592 (patched)
> > 
> >
> > It will be good to not add extra tokens in grammar as it increases the 
> > size of state machine. How about:
> > 
> > 
> > KW_IS (a=KW_NOT)? KW_DISTINCT KW_FROM
> > -> {$a !=null}? ^(EQUAL_NS)
> > -> ^(KW_NOT EQUAL_NS)

I agree and I tried not to add new tokens but I couldn't figure out how to 
write grammar in such a way to avoid adding it. 

For is distinct from we want AST as follows
KW_NOT
  EQUAL_NS
Expr1
Expr2
  
isDistinctFrom rule is invoked from precedenceEqualOperator which is suppose to 
return an AST for operator. This AST is further used by 
precedenceEqualOperator's invoker to make an AST with returned AST as root and 
with two expression as it's children. So if isDistinctFrom return this AST
  KW_NOT
EQUAL_NS

  invoker of precedenceEqualOperator will end up creating
  KW_NOT
   EQUAL_NS
   Expr1
   Expr2
   
  which is not what we want.
  
 Your above suggestion throws an exception while parsing FAILED: 
RewriteEmptyStreamException token KW_NOT.
 
 I am not sure why


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58181/#review171125
---


On April 4, 2017, 11:05 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58181/
> ---
> 
> (Updated April 4, 2017, 11:05 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-15986
> https://issues.apache.org/jira/browse/HIVE-15986
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch adds support for 'is distinct from' and 'is not distinct from'.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 7a70c9c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ccfb455 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  85450c9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8c4ee8a 
>   ql/src/test/queries/clientpositive/is_distinct_from.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/is_distinct_from.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/58181/diff/2/
> 
> 
> Testing
> ---
> 
> Added new tests
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

[jira] [Created] (HIVE-16388) LLAP: Log rotation for daemon, history and gc files

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-16388:


 Summary: LLAP: Log rotation for daemon, history and gc files
 Key: HIVE-16388
 URL: https://issues.apache.org/jira/browse/HIVE-16388
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 3.0.0
Reporter: Siddharth Seth
Assignee: Prasanth Jayachandran


GC logs need to be rotated by date.
LLAP daemon history logs as well
Ideally, the daemon.out file needs the same
Need to be able to download relevant logfiles for a time window.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16387) Fix test failing org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16387:
--

 Summary: Fix test failing 
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
 Key: HIVE-16387
 URL: https://issues.apache.org/jira/browse/HIVE-16387
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16386) Add debug logging to describe why runtime filtering semijoins are removed

[Jason Dere (JIRA)]

Jason Dere created HIVE-16386:
-

 Summary: Add debug logging to describe why runtime filtering 
semijoins are removed
 Key: HIVE-16386
 URL: https://issues.apache.org/jira/browse/HIVE-16386
 Project: Hive
  Issue Type: Improvement
  Components: Logging
Reporter: Jason Dere
Assignee: Jason Dere


Add a few logging statements to detail the reason why semijoin optimizations 
are being removed, which can help during debugging.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16385) StatsNoJobTask could exit early before all partitions have been processed

[Chao Sun (JIRA)]

Chao Sun created HIVE-16385:
---

 Summary: StatsNoJobTask could exit early before all partitions 
have been processed
 Key: HIVE-16385
 URL: https://issues.apache.org/jira/browse/HIVE-16385
 Project: Hive
  Issue Type: Bug
  Components: Statistics
Reporter: Chao Sun
Assignee: Chao Sun


For a partitioned table, the class {{StatsNoJobTask}} is supposed to launch 
threads for all partitions and compute their stats. However, it could exit 
early after at most 100 seconds:

{code}
  private void shutdownAndAwaitTermination(ExecutorService threadPool) {

// Disable new tasks from being submitted
threadPool.shutdown();
try {

  // Wait a while for existing tasks to terminate
  if (!threadPool.awaitTermination(100, TimeUnit.SECONDS)) {
// Cancel currently executing tasks
threadPool.shutdownNow();

// Wait a while for tasks to respond to being cancelled
if (!threadPool.awaitTermination(100, TimeUnit.SECONDS)) {
  LOG.debug("Stats collection thread pool did not terminate");
}
  }
} catch (InterruptedException ie) {

  // Cancel again if current thread also interrupted
  threadPool.shutdownNow();

  // Preserve interrupt status
  Thread.currentThread().interrupt();
}
  }
{code}

The {{shutdown}} call does not wait for all submitted tasks to complete, and 
the {{awaitTermination}} call waits at most 100 seconds. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16384) Remove jdk7 build from travis

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-16384:


 Summary: Remove jdk7 build from travis
 Key: HIVE-16384
 URL: https://issues.apache.org/jira/browse/HIVE-16384
 Project: Hive
  Issue Type: Bug
  Components: Build Infrastructure
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
Priority: Trivial


Since master is updated to JDK8 as minimum required version, remove JDK7 build 
from travis to avoid builds fail showing up in README. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16383) Switch to HikariCP as default connection pooling

[Prasanth Jayachandran (JIRA)]

Prasanth Jayachandran created HIVE-16383:


 Summary: Switch to HikariCP as default connection pooling
 Key: HIVE-16383
 URL: https://issues.apache.org/jira/browse/HIVE-16383
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Affects Versions: 3.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Since 3.0 is planned to move to JDK8, we can now switch to HikariCP as default 
connection pooling for DN because of its improved performance over others. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Peter Vary]

> On April 5, 2017, 4:11 p.m., Sahil Takiar wrote:
> > Minor comments, overall LGTM

Thanks for the reviews Sahil and Vihang!


> On April 5, 2017, 4:11 p.m., Sahil Takiar wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java
> > Lines 57 (patched)
> > 
> >
> > Would be nice to have a test case where the query is exactly the same 
> > as the truncation limit.
> 
> Vihang Karajgaonkar wrote:
> Adding to Sahil's suggestion above. Can we add a test case to make sure 
> that query string is not truncated when config is -1 to make sure we catch 
> regressions in the future

Added a test to test the default value, and when the query is exactly the same 
as the truncation limit, thanks for the suggestion Sahil and Vihang!


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171132
---


On April 5, 2017, 5:21 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> ---
> 
> (Updated April 5, 2017, 5:21 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos 
> Csanady, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query 
> string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new 
> parameter
> - When constructing the HiveLockObject truncates the length of the query 
> string if it is necessary
> - Updates every occurance of the HiveLockObject creation
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6de6ed6 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java 
> a371a5a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
> de3b8ad 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
> 0afbc1c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 
> 19cb129 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
> 
> 
> Diff: https://reviews.apache.org/r/58086/diff/2/
> 
> 
> Testing
> ---
> 
> Added new unit test to test the truncation
> Updated the existing unit tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Peter Vary]

> On April 5, 2017, 4:40 p.m., Vihang Karajgaonkar wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Lines 1774 (patched)
> > 
> >
> > Do you think we should use a large value for the default instead of -1?

Good idea, simplified the code as well.

Thanks,
Peter


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171135
---


On April 5, 2017, 5:21 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> ---
> 
> (Updated April 5, 2017, 5:21 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos 
> Csanady, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query 
> string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new 
> parameter
> - When constructing the HiveLockObject truncates the length of the query 
> string if it is necessary
> - Updates every occurance of the HiveLockObject creation
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6de6ed6 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java 
> a371a5a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
> de3b8ad 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
> 0afbc1c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 
> 19cb129 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
> 
> 
> Diff: https://reviews.apache.org/r/58086/diff/2/
> 
> 
> Testing
> ---
> 
> Added new unit test to test the truncation
> Updated the existing unit tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/
---

(Updated April 5, 2017, 5:21 p.m.)


Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos Csanady, 
and Barna Zsombor Klara.


Changes
---

Changed default value from -1 to Integer.MAX_VALUE as suggested by Vihang.
Added extra test cases for edge cases


Bugs: HIVE-16334
https://issues.apache.org/jira/browse/HIVE-16334


Repository: hive-git


Description
---

The patch contains the following changes:
- Added a new config variable which defines the maximum length of the query 
string in the lock object
- When constructing the HiveLockObject gets the configuration as a new parameter
- When constructing the HiveLockObject truncates the length of the query string 
if it is necessary
- Updates every occurance of the HiveLockObject creation


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6de6ed6 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java a371a5a 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
de3b8ad 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
0afbc1c 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 19cb129 
  
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
 3f9926e 


Diff: https://reviews.apache.org/r/58086/diff/2/

Changes: https://reviews.apache.org/r/58086/diff/1-2/


Testing
---

Added new unit test to test the truncation
Updated the existing unit tests


Thanks,

Peter Vary

[jira] [Created] (HIVE-16382) Find a way to store the query string only once in ZooKeeper locks

[Peter Vary (JIRA)]

Peter Vary created HIVE-16382:
-

 Summary: Find a way to store the query string only once in 
ZooKeeper locks
 Key: HIVE-16382
 URL: https://issues.apache.org/jira/browse/HIVE-16382
 Project: Hive
  Issue Type: Improvement
  Components: Locking
Affects Versions: 3.0.0
Reporter: Peter Vary


Currently when we store the locks in ZooKeeper, then every lock object contains 
the query string. HIVE-16334 aims to lessen the memory pressure on ZooKeeper by 
adding the possibility to store only the truncated version of the query string.

It would be even better to find a solution for storing the string only once, or 
not at all, but still be able to provide it when they are queried by the SHOW 
LOCKS EXTENDED.

cc: [~stakiar]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Vihang Karajgaonkar]

> On April 5, 2017, 4:11 p.m., Sahil Takiar wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java
> > Lines 57 (patched)
> > 
> >
> > Would be nice to have a test case where the query is exactly the same 
> > as the truncation limit.

Adding to Sahil's suggestion above. Can we add a test case to make sure that 
query string is not truncated when config is -1 to make sure we catch 
regressions in the future


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171132
---


On March 30, 2017, 6:01 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> ---
> 
> (Updated March 30, 2017, 6:01 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos 
> Csanady, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query 
> string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new 
> parameter
> - When constructing the HiveLockObject truncates the length of the query 
> string if it is necessary
> - Updates every occurance of the HiveLockObject creation
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e12fea1 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java 
> a371a5a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
> de3b8ad 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
> 0afbc1c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 
> 19cb129 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
> 
> 
> Diff: https://reviews.apache.org/r/58086/diff/1/
> 
> 
> Testing
> ---
> 
> Added new unit test to test the truncation
> Updated the existing unit tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Vihang Karajgaonkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171135
---


Fix it, then Ship it!




Thanks for the patch Peter. Overall looks good. Couple of suggestions below


common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
Lines 1774 (patched)


Do you think we should use a large value for the default instead of -1?


- Vihang Karajgaonkar


On March 30, 2017, 6:01 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> ---
> 
> (Updated March 30, 2017, 6:01 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos 
> Csanady, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query 
> string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new 
> parameter
> - When constructing the HiveLockObject truncates the length of the query 
> string if it is necessary
> - Updates every occurance of the HiveLockObject creation
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e12fea1 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java 
> a371a5a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
> de3b8ad 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
> 0afbc1c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 
> 19cb129 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
> 
> 
> Diff: https://reviews.apache.org/r/58086/diff/1/
> 
> 
> Testing
> ---
> 
> Added new unit test to test the truncation
> Updated the existing unit tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 58086: HIVE-16334 Query lock contains the query string, which can cause OOM on ZooKeeper

[Sahil Takiar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58086/#review171132
---



Minor comments, overall LGTM


ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java
Lines 57 (patched)


Would be nice to have a test case where the query is exactly the same as 
the truncation limit.


- Sahil Takiar


On March 30, 2017, 6:01 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58086/
> ---
> 
> (Updated March 30, 2017, 6:01 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Chaoyu Tang, Marta Kuczora, Miklos 
> Csanady, and Barna Zsombor Klara.
> 
> 
> Bugs: HIVE-16334
> https://issues.apache.org/jira/browse/HIVE-16334
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The patch contains the following changes:
> - Added a new config variable which defines the maximum length of the query 
> string in the lock object
> - When constructing the HiveLockObject gets the configuration as a new 
> parameter
> - When constructing the HiveLockObject truncates the length of the query 
> string if it is necessary
> - Updates every occurance of the HiveLockObject creation
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e12fea1 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DummyTxnManager.java 53ee9c8 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java fff03df 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveTxnManagerImpl.java 
> a371a5a 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDummyTxnManager.java 
> de3b8ad 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java 
> 0afbc1c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestHiveLockObject.java 
> 19cb129 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/zookeeper/TestZookeeperLockManager.java
>  3f9926e 
> 
> 
> Diff: https://reviews.apache.org/r/58086/diff/1/
> 
> 
> Testing
> ---
> 
> Added new unit test to test the truncation
> Updated the existing unit tests
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 58181: HIVE-15986 Support for 'is [NOT] distinct from'

[Ashutosh Chauhan]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58181/#review171125
---




ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g
Lines 590-592 (patched)


It will be good to not add extra tokens in grammar as it increases the size 
of state machine. How about:

KW_IS (a=KW_NOT)? KW_DISTINCT KW_FROM
-> {$a !=null}? ^(EQUAL_NS)
-> ^(KW_NOT EQUAL_NS)


- Ashutosh Chauhan


On April 4, 2017, 11:05 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58181/
> ---
> 
> (Updated April 4, 2017, 11:05 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-15986
> https://issues.apache.org/jira/browse/HIVE-15986
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch adds support for 'is distinct from' and 'is not distinct from'.
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties 7a70c9c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java ccfb455 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/SqlFunctionConverter.java
>  85450c9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8c4ee8a 
>   ql/src/test/queries/clientpositive/is_distinct_from.q PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/is_distinct_from.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/58181/diff/2/
> 
> 
> Testing
> ---
> 
> Added new tests
> Pre-commit testing
> 
> 
> Thanks,
> 
> Vineet Garg
> 
>

[jira] [Created] (HIVE-16381) Flaky test: org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData

[Sahil Takiar (JIRA)]

Sahil Takiar created HIVE-16381:
---

 Summary: Flaky test: 
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
 Key: HIVE-16381
 URL: https://issues.apache.org/jira/browse/HIVE-16381
 Project: Hive
  Issue Type: Sub-task
  Components: Test
Reporter: Sahil Takiar


Exampled Failed Job: 
https://builds.apache.org/job/PreCommit-HIVE-Build/4554/testReport/org.apache.hive.jdbc/TestJdbcDriver2/testResultSetMetaData/

Stack trace:

{code}
org.junit.ComparisonFailure: expected:<[c1]> but was:<[_col0]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData(TestJdbcDriver2.java:1541)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:168)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 58203: HIVE-16345 BeeLineDriver should be able to run qtest files which are using default database tables

[Peter Vary]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58203/
---

Review request for hive, Aihua Xu, Zoltan Haindrich, Yongzhi Chen, and Barna 
Zsombor Klara.


Bugs: HIVE-16345
https://issues.apache.org/jira/browse/HIVE-16345


Repository: hive-git


Description
---

The goal of the change is to run qtest files which contain queries on tables 
created by the init scripts.
It adds the possibility to rewrite the src table references to default.src

This patch contains the following changes:
- Added new parameter to the driver, to control weather the rewrite the table 
names or not (test.rewrite.source.tables) - default is true
- Made QTestUtil.getSrcTables() available for QFile class
- Run the QFile not with "!run testfile.q", but reading the file, and 
assembling the commands - enable us to parse the queries, and provide better 
feedback about the failing queries
- QFile rewrites the source tables, if it is required
- Used 9 qtest files from the CliDriver, and added them to BeeLine tests
- Added new filters, and removed redundant ones - I was able to remove every 
QFile specific filter, and corresponding setter methods as well
- Moved QFile classes to org.apache.hive.beeline package, so it can use package 
private methods from BeeLine, and Commands
- Refactored needsContinuation method in BeeLine, so it can be called from a 
static context as well

And one important change is:
- In Utilities.setMapRedWork, change the INPUT_NAME value in the conf to a 
mapreduce task specific value. This one is used by the IOContextMap to cache 
the IOContext objects. Using the same value for every mapred task prevented 
them to run in the same JVM. The test were running sequencially, but failed 
randomly in parallel


Diffs
-

  beeline/src/java/org/apache/hive/beeline/BeeLine.java 11526a7 
  itests/src/test/resources/testconfiguration.properties 7a70c9c 
  
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java
 0d63f5d 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 2abf252 
  itests/util/src/main/java/org/apache/hive/beeline/qfile/QFile.java ae5a349 
  
itests/util/src/main/java/org/apache/hive/beeline/qfile/QFileBeeLineClient.java 
760fde6 
  itests/util/src/main/java/org/apache/hive/beeline/qfile/package-info.java 
fcd50ec 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 79955e9 
  ql/src/test/results/clientpositive/beeline/drop_with_concurrency.q.out 
385f9b7 
  ql/src/test/results/clientpositive/beeline/escape_comments.q.out abc0fee 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_10.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_11.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_12.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_13.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_16.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_3.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/58203/diff/1/


Testing
---

Run the test multiple times with the various combinations of the following 
parameters:
- test.rewrite.source.tables - runs with true, or without it, fails when set to 
false
- junit.parallel.threads - runs with 1, or without this parameter


Thanks,

Peter Vary