[jira] [Commented] (HIVE-15936) ConcurrentModificationException in ATSHook

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869446#comment-15869446
 ] 

Hive QA commented on HIVE-15936:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852954/HIVE-15936.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3590/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3590/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3590/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-02-16 07:58:04.403
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3590/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-02-16 07:58:04.405
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b63f7d7 HIVE-15892: Vectorization: Fast Hash tables need to do 
bounds checking during expand (Matt McCline, reviewed by Jason Dere
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b63f7d7 HIVE-15892: Vectorization: Fast Hash tables need to do 
bounds checking during expand (Matt McCline, reviewed by Jason Dere
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-02-16 07:58:05.274
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java: No such file 
or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852954 - PreCommit-HIVE-Build

> ConcurrentModificationException in ATSHook
> --
>
> Key: HIVE-15936
> URL: https://issues.apache.org/jira/browse/HIVE-15936
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15936.1.patch, HIVE-15936.2.patch
>
>
> See ATSHook error:
> {noformat}
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
> ~[?:1.8.0_112]
>   at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
> ~[?:1.8.0_112]
>   at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:290) 
> ~[guava-14.0.1.jar:?]
>   at 
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:219) 
> ~[hive-common-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> org.apache.hadoop.hive.ql.hooks.ATSHook.createPostHookEvent(ATSHook.java:347) 
> ~[hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:206) 
> [hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [?:1.8.0_112]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> According to [~jdere], ATSHook is currently accessing the PerfLogger on 

[jira] [Commented] (HIVE-15935) ACL is not set in ATS data

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869439#comment-15869439
 ] 

Hive QA commented on HIVE-15935:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852942/HIVE-15935.1.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3588/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3588/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3588/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-02-16 07:56:56.402
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3588/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-02-16 07:56:56.405
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b63f7d7 HIVE-15892: Vectorization: Fast Hash tables need to do 
bounds checking during expand (Matt McCline, reviewed by Jason Dere
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b63f7d7 HIVE-15892: Vectorization: Fast Hash tables need to do 
bounds checking during expand (Matt McCline, reviewed by Jason Dere
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-02-16 07:56:57.490
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: No such 
file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java: No such 
file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java: No such file 
or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java: No 
such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852942 - PreCommit-HIVE-Build

> ACL is not set in ATS data
> --
>
> Key: HIVE-15935
> URL: https://issues.apache.org/jira/browse/HIVE-15935
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15935.1.patch
>
>
> When publishing ATS info, Hive does not set ACL, that make Hive ATS entries 
> visible to all users. On the other hand, Tez ATS entires is using Tez DAG ACL 
> which limit both view/modify ACL to end user only. We shall make them 
> consistent. In the Jira, I am going to limit ACL to end user for both Tez ATS 
> and Hive ATS, also provide config "hive.view.acls" and "hive.modify.acls" if 
> user need to overridden.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15936) ConcurrentModificationException in ATSHook

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869442#comment-15869442
 ] 

Hive QA commented on HIVE-15936:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852954/HIVE-15936.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3589/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3589/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3589/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-02-16 07:57:31.303
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3589/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-02-16 07:57:31.306
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b63f7d7 HIVE-15892: Vectorization: Fast Hash tables need to do 
bounds checking during expand (Matt McCline, reviewed by Jason Dere
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b63f7d7 HIVE-15892: Vectorization: Fast Hash tables need to do 
bounds checking during expand (Matt McCline, reviewed by Jason Dere
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-02-16 07:57:32.278
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java: No such file 
or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852954 - PreCommit-HIVE-Build

> ConcurrentModificationException in ATSHook
> --
>
> Key: HIVE-15936
> URL: https://issues.apache.org/jira/browse/HIVE-15936
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15936.1.patch, HIVE-15936.2.patch
>
>
> See ATSHook error:
> {noformat}
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
> ~[?:1.8.0_112]
>   at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
> ~[?:1.8.0_112]
>   at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:290) 
> ~[guava-14.0.1.jar:?]
>   at 
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:219) 
> ~[hive-common-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> org.apache.hadoop.hive.ql.hooks.ATSHook.createPostHookEvent(ATSHook.java:347) 
> ~[hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:206) 
> [hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [?:1.8.0_112]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> According to [~jdere], ATSHook is currently accessing the PerfLogger on 

[jira] [Commented] (HIVE-15919) Row count mismatch for count * query

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869437#comment-15869437
 ] 

Hive QA commented on HIVE-15919:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852940/HIVE-15919.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 10233 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_mapjoin] 
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_reduce_groupby_decimal]
 (batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_13] 
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_8] 
(batchId=43)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_mapjoin]
 (batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_decimal]
 (batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_13]
 (batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_8]
 (batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_short_regress]
 (batchId=148)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31] 
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_mapjoin]
 (batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_13] 
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress]
 (batchId=115)
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
 (batchId=170)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3587/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3587/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3587/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852940 - PreCommit-HIVE-Build

> Row count mismatch for count * query
> 
>
> Key: HIVE-15919
> URL: https://issues.apache.org/jira/browse/HIVE-15919
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Jason Dere
> Attachments: HIVE-15919.1.patch, table_14.q, table_6.q
>
>
> The following query is returning different output when run against hive and 
> postgres.
> Query:
> SELECT COUNT (*)
> FROM
> (SELECT LAG(COALESCE(t2.int_col_14, t1.int_col_80),22) OVER (ORDER BY 
> t1.tinyint_col_52 DESC) AS int_col
> FROM table_6 t1
> INNER JOIN table_14 t2 ON ((t2.decimal0101_col_55) = (t1.decimal0101_col_9))) 
> AS FOO;
> From hive: 0
> From postgres: 66903279
> Attaching ddl files for the tables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15910) Improvements in Hive Unit Test by using In-memory Derby DB

[Sankar Hariappan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sankar Hariappan updated HIVE-15910:

Attachment: HIVE-15910.05.patch

Fixed some cleanup issues in TestWorker test cases.

> Improvements in Hive Unit Test by using In-memory Derby DB
> --
>
> Key: HIVE-15910
> URL: https://issues.apache.org/jira/browse/HIVE-15910
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Sankar Hariappan
>Assignee: Wei Zheng
> Attachments: HIVE-15910.01.patch, HIVE-15910.05.patch, 
> HIVE-15910.2.patch, HIVE-15910.3.patch, HIVE-15910.4.patch
>
>
> Hive UT currently uses Derby DB with storage on disk which have some 
> practical problems.
> 1. The run-time of Hive unit tests are high as need to operate on the disk 
> quite often.
> 2. It can cause conflict if multiple test cases operates on the same table 
> name (such as table being created already exist).
> To solve these problems, we shall use an in-memory storage option of Derby DB 
> which can be even persisted if the test case demands that.
> https://db.apache.org/derby/docs/10.8/devguide/cdevdvlpinmemdb.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15161) migrate ColumnStats to use jackson

[Pengcheng Xiong (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869403#comment-15869403
 ] 

Pengcheng Xiong commented on HIVE-15161:


LGTM +1. Did u run the QA tests already? Thanks.

> migrate ColumnStats to use jackson
> --
>
> Key: HIVE-15161
> URL: https://issues.apache.org/jira/browse/HIVE-15161
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Fix For: 2.2.0
>
> Attachments: HIVE-15161.1.patch, HIVE-15161.2.patch, 
> HIVE-15161.3.patch, HIVE-15161.4.patch, HIVE-15161.4.patch, 
> HIVE-15161.5.patch, HIVE-15161.5.patch, HIVE-15161.6.patch
>
>
> * json.org has license issues
> * jackson can provide a fully compatible alternative to it
> * there are a few flakiness issues caused by the order of the map entries of 
> the columns...this cat be addressed, org.json api was unfriendly in this 
> manner ;)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15933) Improve plans for correlated subquery with join and predicate

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869371#comment-15869371
 ] 

Hive QA commented on HIVE-15933:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852935/HIVE-15933.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 10229 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
TestSSL - did not produce a TEST-*.xml file (likely timed out) (batchId=214)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucket2] 
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby8_map_skew] 
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[identity_project_remove_skip]
 (batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31] 
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_subquery] 
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[nullgroup2] 
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_1] 
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats5] 
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_8] 
(batchId=116)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3586/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3586/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3586/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852935 - PreCommit-HIVE-Build

> Improve plans for correlated subquery with join and predicate
> -
>
> Key: HIVE-15933
> URL: https://issues.apache.org/jira/browse/HIVE-15933
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-15933.1.patch
>
>
> This is a continuation of HIVE-15905
> for queries such as:
> {code:SQL}
> explain select  
>   cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   count(*) cnt1,
>   cd_purchase_estimate,
>   count(*) cnt2,
>   cd_credit_rating,
>   count(*) cnt3,
>   cd_dep_count,
>   count(*) cnt4,
>   cd_dep_employed_count,
>   count(*) cnt5,
>   cd_dep_college_count,
>   count(*) cnt6
>  from
>   customer c,customer_address ca,customer_demographics
>  where
>   c.c_current_addr_sk = ca.ca_address_sk and
>   ca_county in ('Walker County','Richland County','Gaines County','Douglas 
> County','Dona Ana County') and
>   cd_demo_sk = c.c_current_cdemo_sk and 
>   exists (select *
>   from store_sales,date_dim
>   where c.c_customer_sk = ss_customer_sk and
> ss_sold_date_sk = d_date_sk and
> d_year = 2002 and
> d_moy between 4 and 4+3)
>  group by cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   cd_purchase_estimate,
>   cd_credit_rating,
>   cd_dep_count,
>   cd_dep_employed_count,
>   cd_dep_college_count
>  order by cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   cd_purchase_estimate,
>   cd_credit_rating,
>   cd_dep_count,
>   cd_dep_employed_count,
>   cd_dep_college_count
> limit 100;
> {code}
> HIVE generates un-necessary joins to produce value for correlated columns.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15934) Downgrade Maven surefire plugin from 2.19.1 to 2.18.1

[Zoltan Haindrich (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869361#comment-15869361
 ] 

Zoltan Haindrich commented on HIVE-15934:
-

[~wzheng] thank you for opening this...I didn't had time to look into it - 
sorry for that...

> Downgrade Maven surefire plugin from 2.19.1 to 2.18.1
> -
>
> Key: HIVE-15934
> URL: https://issues.apache.org/jira/browse/HIVE-15934
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15934.1.patch
>
>
> Surefire 2.19.1 has some issue 
> (https://issues.apache.org/jira/browse/SUREFIRE-1255) which caused debugging 
> session to abort after a short period of time. Many IntelliJ users have seen 
> this, although it looks fine for Eclipse users. Version 2.18.1 works fine.
> We'd better make the change to not impact the development for IntelliJ guys. 
> We can upgrade again once the root cause is figured out.
> cc [~kgyrtkirk] [~ashutoshc]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-6590) Hive does not work properly with boolean partition columns (wrong results and inserts to incorrect HDFS path)

[Zoltan Haindrich (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869325#comment-15869325
 ] 

Zoltan Haindrich commented on HIVE-6590:


partition keys are parsed here:
https://github.com/apache/hive/blob/d357f38521ae583007ff96ed7090ac41f56b78b2/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java#L257

I was expecting a whole myriad of qtests to fail when I submitted the patch the 
first time...but there were only just a few - how much does the fact that 
something is "under-tested" correlates with its being "under-used" ? :)

Anyway; of course its possible to apply local changes to the partition key 
parsing code...but I think there is an alternative path:

* parse empty strings and "false" as false (optionally ignoring casing)
* all others as true

this would be a minimal change to the serde code which would be enough to fix 
the partition parsing.

about cast: I haven't looked into sql2011 specs about this aspect, but i'm 
pretty confident that it will suggest the following {{cast('false' as 
boolean)}} should be false...

> Hive does not work properly with boolean partition columns (wrong results and 
> inserts to incorrect HDFS path)
> -
>
> Key: HIVE-6590
> URL: https://issues.apache.org/jira/browse/HIVE-6590
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Metastore
>Affects Versions: 0.10.0
>Reporter: Lenni Kuff
>Assignee: Zoltan Haindrich
> Attachments: HIVE-6590.1.patch, HIVE-6590.2.patch, HIVE-6590.3.patch
>
>
> Hive does not work properly with boolean partition columns. Queries return 
> wrong results and also insert to incorrect HDFS paths.
> {code}
> create table bool_part(int_col int) partitioned by(bool_col boolean);
> # This works, creating 3 unique partitions!
> ALTER TABLE bool_table ADD PARTITION (bool_col=FALSE);
> ALTER TABLE bool_table ADD PARTITION (bool_col=false);
> ALTER TABLE bool_table ADD PARTITION (bool_col=False);
> {code}
> The first problem is that Hive cannot filter on a bool partition key column. 
> "select * from bool_part" returns the correct results, but if you apply a 
> filter on the bool partition key column hive won't return any results.
> The second problem is that Hive seems to just call "toString()" on the 
> boolean literal value. This means you can end up with multiple partitions 
> (FALSE, false, FaLSE, etc) mapping to the literal value 'FALSE'. For example, 
> if you can add three partition in have for the same logic value "false" doing:
> ALTER TABLE bool_table ADD PARTITION (bool_col=FALSE) -> 
> /test-warehouse/bool_table/bool_col=FALSE/
> ALTER TABLE bool_table ADD PARTITION (bool_col=false) -> 
> /test-warehouse/bool_table/bool_col=false/
> ALTER TABLE bool_table ADD PARTITION (bool_col=False) -> 
> /test-warehouse/bool_table/bool_col=False/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-1010:
-
Attachment: HIVE-1010.2.patch

> Implement INFORMATION_SCHEMA in Hive
> 
>
> Key: HIVE-1010
> URL: https://issues.apache.org/jira/browse/HIVE-1010
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Query Processor, Server Infrastructure
>Reporter: Jeff Hammerbacher
>Assignee: Gunther Hagleitner
> Attachments: HIVE-1010.1.patch, HIVE-1010.2.patch
>
>
> INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to 
> implement using our metastore.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-1010:
-
Attachment: (was: HIVE-1010.2.patch)

> Implement INFORMATION_SCHEMA in Hive
> 
>
> Key: HIVE-1010
> URL: https://issues.apache.org/jira/browse/HIVE-1010
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Query Processor, Server Infrastructure
>Reporter: Jeff Hammerbacher
>Assignee: Gunther Hagleitner
> Attachments: HIVE-1010.1.patch, HIVE-1010.2.patch
>
>
> INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to 
> implement using our metastore.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-1010) Implement INFORMATION_SCHEMA in Hive

[Gunther Hagleitner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-1010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gunther Hagleitner updated HIVE-1010:
-
Attachment: HIVE-1010.2.patch

> Implement INFORMATION_SCHEMA in Hive
> 
>
> Key: HIVE-1010
> URL: https://issues.apache.org/jira/browse/HIVE-1010
> Project: Hive
>  Issue Type: New Feature
>  Components: Metastore, Query Processor, Server Infrastructure
>Reporter: Jeff Hammerbacher
>Assignee: Gunther Hagleitner
> Attachments: HIVE-1010.1.patch, HIVE-1010.2.patch
>
>
> INFORMATION_SCHEMA is part of the SQL92 standard and would be useful to 
> implement using our metastore.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15934) Downgrade Maven surefire plugin from 2.19.1 to 2.18.1

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869302#comment-15869302
 ] 

Hive QA commented on HIVE-15934:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852937/HIVE-15934.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10238 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31] 
(batchId=133)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3585/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3585/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3585/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852937 - PreCommit-HIVE-Build

> Downgrade Maven surefire plugin from 2.19.1 to 2.18.1
> -
>
> Key: HIVE-15934
> URL: https://issues.apache.org/jira/browse/HIVE-15934
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15934.1.patch
>
>
> Surefire 2.19.1 has some issue 
> (https://issues.apache.org/jira/browse/SUREFIRE-1255) which caused debugging 
> session to abort after a short period of time. Many IntelliJ users have seen 
> this, although it looks fine for Eclipse users. Version 2.18.1 works fine.
> We'd better make the change to not impact the development for IntelliJ guys. 
> We can upgrade again once the root cause is figured out.
> cc [~kgyrtkirk] [~ashutoshc]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background

[Ferdinand Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869301#comment-15869301
 ] 

Ferdinand Xu commented on HIVE-15710:
-

LGTM +1 pending on the test

> HS2 Stopped when running in background
> --
>
> Key: HIVE-15710
> URL: https://issues.apache.org/jira/browse/HIVE-15710
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15710.1.patch, HIVE-15710.2.patch, 
> HIVE-15710.3.patch
>
>
> To reproduce, start HS2 in background like {{hive --service hiveserver2 &}}, 
> and run some query from beeline using Tez or Spark as engine.
> I think it's similar to HIVE-6758: HS2 uses jline for the in-place progress 
> update, and receives SIGTTOU when trying to call stty.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15710) HS2 Stopped when running in background

[Rui Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rui Li updated HIVE-15710:
--
Attachment: HIVE-15710.3.patch

Update to address comments. I think CLI also needs this logic. [~Ferd] let me 
know if you think otherwise.

> HS2 Stopped when running in background
> --
>
> Key: HIVE-15710
> URL: https://issues.apache.org/jira/browse/HIVE-15710
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15710.1.patch, HIVE-15710.2.patch, 
> HIVE-15710.3.patch
>
>
> To reproduce, start HS2 in background like {{hive --service hiveserver2 &}}, 
> and run some query from beeline using Tez or Spark as engine.
> I think it's similar to HIVE-6758: HS2 uses jline for the in-place progress 
> update, and receives SIGTTOU when trying to call stty.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15931) JDBC: Improve logging when using ZooKeeper

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869222#comment-15869222
 ] 

Hive QA commented on HIVE-15931:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852931/HIVE-15931.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10238 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31] 
(batchId=133)
org.apache.hive.jdbc.TestJdbcDriver2.testBadURL (batchId=215)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3584/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3584/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3584/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852931 - PreCommit-HIVE-Build

> JDBC: Improve logging when using ZooKeeper
> --
>
> Key: HIVE-15931
> URL: https://issues.apache.org/jira/browse/HIVE-15931
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15931.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15900) Beeline prints tez job progress in stdout instead of stderr

[anishek (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869203#comment-15869203
 ] 

anishek commented on HIVE-15900:


+1


> Beeline prints tez job progress in stdout instead of stderr
> ---
>
> Key: HIVE-15900
> URL: https://issues.apache.org/jira/browse/HIVE-15900
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Thejas M Nair
> Attachments: HIVE-15900.1.patch, HIVE-15900.2.patch, 
> HIVE-15900.3.patch, HIVE-15900.3.patch, std_out
>
>
> Tez job progress messages are getting updated to stdout instead of stderr
> Attaching output file for below command, with the tez job status printed
> $HIVE_HOME/bin/beeline -n  -p  -u " --outputformat=tsv -e "analyze table studenttab10k compute statistics;" > 
> stdout



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-10562) Add versioning/format mechanism to NOTIFICATION_LOG entries, expand MESSAGE size

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869171#comment-15869171
 ] 

Thejas M Nair commented on HIVE-10562:
--

[~spena]
For some reason the tests for metastore schema upgrade in HIVE-9800 didn't get 
kicked off in this case. Any idea why ?


> Add versioning/format mechanism to NOTIFICATION_LOG entries, expand MESSAGE 
> size
> 
>
> Key: HIVE-10562
> URL: https://issues.apache.org/jira/browse/HIVE-10562
> Project: Hive
>  Issue Type: Sub-task
>  Components: Import/Export
>Affects Versions: 1.2.0
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Fix For: 2.2.0
>
> Attachments: HIVE-10562.2.patch, HIVE-10562.3.patch, 
> HIVE-10562.4.patch, HIVE-10562.5.patch, HIVE-10562.patch
>
>
> Currently, we have a JSON encoded message being stored in the 
> NOTIFICATION_LOG table.
> If we want to be future proof, we need to allow for versioning of this 
> message, since we might change what gets stored in the message. A prime 
> example of what we'd want to change is as in HIVE-10393.
> MessageFactory already has stubs to allow for versioning of messages, and we 
> could expand on this further in the future. NotificationListener currently 
> encodes the message version into the header for the JMS message it sends, 
> which seems to be the right place for a message version (instead of being 
> contained in the message, for eg.).
> So, we should have a similar ability for DbEventListener as well, and the 
> place this makes the most sense is to and add a version column to the 
> NOTIFICATION_LOG table.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15846) Relocate more dependencies (e.g. org.apache.zookeeper) for JDBC uber jar

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869159#comment-15869159
 ] 

Thejas M Nair commented on HIVE-15846:
--

[~taoli-hwx]
The patch isn't applying on master anymore. Can you please take a look ?


> Relocate more dependencies (e.g. org.apache.zookeeper) for JDBC uber jar
> 
>
> Key: HIVE-15846
> URL: https://issues.apache.org/jira/browse/HIVE-15846
> Project: Hive
>  Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-15846.1.patch, HIVE-15846.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15931) JDBC: Improve logging when using ZooKeeper

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869156#comment-15869156
 ] 

Thejas M Nair commented on HIVE-15931:
--

[~vgumashta] Can you please include a reviewboard link or github pull request ?


> JDBC: Improve logging when using ZooKeeper
> --
>
> Key: HIVE-15931
> URL: https://issues.apache.org/jira/browse/HIVE-15931
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15931.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15918) Add some debug messages to identify an issue getting runtimeInfo from tez

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869143#comment-15869143
 ] 

Prasanth Jayachandran commented on HIVE-15918:
--

+1 for the new changes

> Add some debug messages to identify an issue getting runtimeInfo from tez
> -
>
> Key: HIVE-15918
> URL: https://issues.apache.org/jira/browse/HIVE-15918
> Project: Hive
>  Issue Type: Task
>  Components: llap
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-15918.01.patch, HIVE-15918.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15918) Add some debug messages to identify an issue getting runtimeInfo from tez

[Siddharth Seth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated HIVE-15918:
--
Attachment: HIVE-15918.02.patch

Added one more log line in LlapTaskSchedulerService. [~prasanth_j] - could you 
take another look please.

> Add some debug messages to identify an issue getting runtimeInfo from tez
> -
>
> Key: HIVE-15918
> URL: https://issues.apache.org/jira/browse/HIVE-15918
> Project: Hive
>  Issue Type: Task
>  Components: llap
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-15918.01.patch, HIVE-15918.02.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15841) Upgrade Hive to ORC 1.3.2

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869135#comment-15869135
 ] 

Hive QA commented on HIVE-15841:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852928/HIVE-15841.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10238 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
 (batchId=148)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join1]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_outer_join2]
 (batchId=160)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31] 
(batchId=133)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3583/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3583/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3583/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852928 - PreCommit-HIVE-Build

> Upgrade Hive to ORC 1.3.2
> -
>
> Key: HIVE-15841
> URL: https://issues.apache.org/jira/browse/HIVE-15841
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-15841.patch
>
>
> Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.2 once it 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15891) Detect query rewrite scenario for UPDATE/DELETE/MERGE and fail fast

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869118#comment-15869118
 ] 

Wei Zheng commented on HIVE-15891:
--

It fails for all merge queries

> Detect query rewrite scenario for UPDATE/DELETE/MERGE and fail fast
> ---
>
> Key: HIVE-15891
> URL: https://issues.apache.org/jira/browse/HIVE-15891
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15891.1.patch, HIVE-15891.2.patch
>
>
> Currently ACID UpdateDeleteSemanticAnalyzer directly manipulates the AST tree 
> but it's different from the general approach of modifying the token stream 
> and thus will cause AST tree mismatch if there is any rewrite happening after 
> UpdateDeleteSemanticAnalyzer.
> The long term solution will be to rewrite the AST handling logic in 
> UpdateDeleteSemanticAnalyzer, to make it consistent with the general approach.
> This ticket will for now detect the error prone cases and fail early. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15900) Beeline prints tez job progress in stdout instead of stderr

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869119#comment-15869119
 ] 

Thejas M Nair commented on HIVE-15900:
--

Attaching same file again, many of the failed test pass locally.


> Beeline prints tez job progress in stdout instead of stderr
> ---
>
> Key: HIVE-15900
> URL: https://issues.apache.org/jira/browse/HIVE-15900
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Thejas M Nair
> Attachments: HIVE-15900.1.patch, HIVE-15900.2.patch, 
> HIVE-15900.3.patch, HIVE-15900.3.patch, std_out
>
>
> Tez job progress messages are getting updated to stdout instead of stderr
> Attaching output file for below command, with the tez job status printed
> $HIVE_HOME/bin/beeline -n  -p  -u " --outputformat=tsv -e "analyze table studenttab10k compute statistics;" > 
> stdout



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15900) Beeline prints tez job progress in stdout instead of stderr

[Thejas M Nair (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thejas M Nair updated HIVE-15900:
-
Attachment: HIVE-15900.3.patch

> Beeline prints tez job progress in stdout instead of stderr
> ---
>
> Key: HIVE-15900
> URL: https://issues.apache.org/jira/browse/HIVE-15900
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Thejas M Nair
> Attachments: HIVE-15900.1.patch, HIVE-15900.2.patch, 
> HIVE-15900.3.patch, HIVE-15900.3.patch, std_out
>
>
> Tez job progress messages are getting updated to stdout instead of stderr
> Attaching output file for below command, with the tez job status printed
> $HIVE_HOME/bin/beeline -n  -p  -u " --outputformat=tsv -e "analyze table studenttab10k compute statistics;" > 
> stdout



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-12631) LLAP: support ORC ACID tables

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-12631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi updated HIVE-12631:
--
Attachment: HIVE-12631.3.patch

This third patch applied Sergey's feedbacks.

1. It uses a Configuration object given in a constructor, not a method argument.
2. It reads delta file only per split, not per batch.
3. It doesn't move the inner classes and methods, to show what are changed.
4. It removed confusing comments.

> LLAP: support ORC ACID tables
> -
>
> Key: HIVE-12631
> URL: https://issues.apache.org/jira/browse/HIVE-12631
> Project: Hive
>  Issue Type: Bug
>  Components: llap, Transactions
>Reporter: Sergey Shelukhin
>Assignee: Teddy Choi
> Attachments: HIVE-12631.1.patch, HIVE-12631.2.patch, 
> HIVE-12631.3.patch
>
>
> LLAP uses a completely separate read path in ORC to allow for caching and 
> parallelization of reads and processing. This path does not support ACID. As 
> far as I remember ACID logic is embedded inside ORC format; we need to 
> refactor it to be on top of some interface, if practical; or just port it to 
> LLAP read path.
> Another consideration is how the logic will work with cache. The cache is 
> currently low-level (CB-level in ORC), so we could just use it to read bases 
> and deltas (deltas should be cached with higher priority) and merge as usual. 
> We could also cache merged representation in future.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (HIVE-15938) position alias in order by fails for union queries

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869072#comment-15869072
 ] 

Sergey Shelukhin edited comment on HIVE-15938 at 2/16/17 3:31 AM:
--

Preliminary patch, may need some cleanup. Also need to see if it breaks tests.
I was actually considering explicitly setting aliases for all SELEXPR right 
after parse, if not set, to avoid this problem at any later point. But for now 
this is it.


was (Author: sershe):
Preliminary patch, may need some cleanup. Also need to see if it breaks tests

> position alias in order by fails for union queries
> --
>
> Key: HIVE-15938
> URL: https://issues.apache.org/jira/browse/HIVE-15938
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15938.patch
>
>
> The problem is that the query introduces a spurious ALLCOLREF



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15938) position alias in order by fails for union queries

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869074#comment-15869074
 ] 

Sergey Shelukhin commented on HIVE-15938:
-

[~ashutoshc] can you take a look?

> position alias in order by fails for union queries
> --
>
> Key: HIVE-15938
> URL: https://issues.apache.org/jira/browse/HIVE-15938
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15938.patch
>
>
> The problem is that the query introduces a spurious ALLCOLREF



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15938) position alias in order by fails for union queries

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15938:

Description: The problem is that the query introduces a spurious ALLCOLREF

> position alias in order by fails for union queries
> --
>
> Key: HIVE-15938
> URL: https://issues.apache.org/jira/browse/HIVE-15938
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15938.patch
>
>
> The problem is that the query introduces a spurious ALLCOLREF



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15938) position alias in order by fails for union queries

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15938:

Status: Patch Available  (was: Open)

> position alias in order by fails for union queries
> --
>
> Key: HIVE-15938
> URL: https://issues.apache.org/jira/browse/HIVE-15938
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15938.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15938) position alias in order by fails for union queries

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15938:

Attachment: HIVE-15938.patch

Preliminary patch, may need some cleanup. Also need to see if it breaks tests

> position alias in order by fails for union queries
> --
>
> Key: HIVE-15938
> URL: https://issues.apache.org/jira/browse/HIVE-15938
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15938.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15938) position alias in order by fails for union queries

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-15938:
---


> position alias in order by fails for union queries
> --
>
> Key: HIVE-15938
> URL: https://issues.apache.org/jira/browse/HIVE-15938
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15910) Improvements in Hive Unit Test by using In-memory Derby DB

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869058#comment-15869058
 ] 

Hive QA commented on HIVE-15910:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852925/HIVE-15910.4.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10238 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31] 
(batchId=133)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorPartitionWithBase 
(batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorPartitionWithBaseMissingBuckets
 (batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableNoBase 
(batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableWithBase 
(batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorPartitionWithBase 
(batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorTableLegacy 
(batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorTableNoBase 
(batchId=246)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3582/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3582/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3582/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852925 - PreCommit-HIVE-Build

> Improvements in Hive Unit Test by using In-memory Derby DB
> --
>
> Key: HIVE-15910
> URL: https://issues.apache.org/jira/browse/HIVE-15910
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Sankar Hariappan
>Assignee: Wei Zheng
> Attachments: HIVE-15910.01.patch, HIVE-15910.2.patch, 
> HIVE-15910.3.patch, HIVE-15910.4.patch
>
>
> Hive UT currently uses Derby DB with storage on disk which have some 
> practical problems.
> 1. The run-time of Hive unit tests are high as need to operate on the disk 
> quite often.
> 2. It can cause conflict if multiple test cases operates on the same table 
> name (such as table being created already exist).
> To solve these problems, we shall use an in-memory storage option of Derby DB 
> which can be even persisted if the test case demands that.
> https://db.apache.org/derby/docs/10.8/devguide/cdevdvlpinmemdb.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15936) ConcurrentModificationException in ATSHook

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869052#comment-15869052
 ] 

Jason Dere commented on HIVE-15936:
---

+1

> ConcurrentModificationException in ATSHook
> --
>
> Key: HIVE-15936
> URL: https://issues.apache.org/jira/browse/HIVE-15936
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15936.1.patch, HIVE-15936.2.patch
>
>
> See ATSHook error:
> {noformat}
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
> ~[?:1.8.0_112]
>   at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
> ~[?:1.8.0_112]
>   at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:290) 
> ~[guava-14.0.1.jar:?]
>   at 
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:219) 
> ~[hive-common-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> org.apache.hadoop.hive.ql.hooks.ATSHook.createPostHookEvent(ATSHook.java:347) 
> ~[hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:206) 
> [hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [?:1.8.0_112]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> According to [~jdere], ATSHook is currently accessing the PerfLogger on a 
> separate thread, which means the main query thread can potentially write to 
> the PerfLogger at the same time.
> The ATSHook should access the PerfLogger on the main query thread, before it 
> sends the execution to the ATS Logger thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15937) RetryingMetaStoreClient.invoke() should retry Trift exception even if not wrapped in MetaException

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-15937:
--
Attachment: HIVE-15937.01.patch

> RetryingMetaStoreClient.invoke() should retry Trift exception even if not 
> wrapped in MetaException
> --
>
> Key: HIVE-15937
> URL: https://issues.apache.org/jira/browse/HIVE-15937
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15937.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15937) RetryingMetaStoreClient.invoke() should retry Trift exception even if not wrapped in MetaException

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-15937:
--
Status: Patch Available  (was: Open)

> RetryingMetaStoreClient.invoke() should retry Trift exception even if not 
> wrapped in MetaException
> --
>
> Key: HIVE-15937
> URL: https://issues.apache.org/jira/browse/HIVE-15937
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15937.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15715) RetryingMetaStoreClient is retrying operations too aggressievley

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-15715:
--
Attachment: (was: HIVE-15715.01.patch)

> RetryingMetaStoreClient is retrying operations too aggressievley
> 
>
> Key: HIVE-15715
> URL: https://issues.apache.org/jira/browse/HIVE-15715
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> RetryingMetaStoreClient is retrying operations w/o paying attention to 
> whether the operation is safe to retry.
> common/src/java/org/apache/hadoop/hive/common/classification/RetrySemantics.java
>  should be used to annotate metastore ops with their retry semantics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15715) RetryingMetaStoreClient is retrying operations too aggressievley

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-15715:
--
Status: Open  (was: Patch Available)

> RetryingMetaStoreClient is retrying operations too aggressievley
> 
>
> Key: HIVE-15715
> URL: https://issues.apache.org/jira/browse/HIVE-15715
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> RetryingMetaStoreClient is retrying operations w/o paying attention to 
> whether the operation is safe to retry.
> common/src/java/org/apache/hadoop/hive/common/classification/RetrySemantics.java
>  should be used to annotate metastore ops with their retry semantics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15715) RetryingMetaStoreClient is retrying operations too aggressievley

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-15715:
-

Assignee: (was: Eugene Koifman)

> RetryingMetaStoreClient is retrying operations too aggressievley
> 
>
> Key: HIVE-15715
> URL: https://issues.apache.org/jira/browse/HIVE-15715
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>
> RetryingMetaStoreClient is retrying operations w/o paying attention to 
> whether the operation is safe to retry.
> common/src/java/org/apache/hadoop/hive/common/classification/RetrySemantics.java
>  should be used to annotate metastore ops with their retry semantics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15715) RetryingMetaStoreClient is retrying operations too aggressievley

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-15715:
--
Attachment: HIVE-15715.01.patch

> RetryingMetaStoreClient is retrying operations too aggressievley
> 
>
> Key: HIVE-15715
> URL: https://issues.apache.org/jira/browse/HIVE-15715
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15715.01.patch
>
>
> RetryingMetaStoreClient is retrying operations w/o paying attention to 
> whether the operation is safe to retry.
> common/src/java/org/apache/hadoop/hive/common/classification/RetrySemantics.java
>  should be used to annotate metastore ops with their retry semantics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15715) RetryingMetaStoreClient is retrying operations too aggressievley

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-15715:
--
Status: Patch Available  (was: Open)

> RetryingMetaStoreClient is retrying operations too aggressievley
> 
>
> Key: HIVE-15715
> URL: https://issues.apache.org/jira/browse/HIVE-15715
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-15715.01.patch
>
>
> RetryingMetaStoreClient is retrying operations w/o paying attention to 
> whether the operation is safe to retry.
> common/src/java/org/apache/hadoop/hive/common/classification/RetrySemantics.java
>  should be used to annotate metastore ops with their retry semantics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15715) RetryingMetaStoreClient is retrying operations too aggressievley

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-15715:
-

Assignee: Eugene Koifman

> RetryingMetaStoreClient is retrying operations too aggressievley
> 
>
> Key: HIVE-15715
> URL: https://issues.apache.org/jira/browse/HIVE-15715
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> RetryingMetaStoreClient is retrying operations w/o paying attention to 
> whether the operation is safe to retry.
> common/src/java/org/apache/hadoop/hive/common/classification/RetrySemantics.java
>  should be used to annotate metastore ops with their retry semantics.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15937) RetryingMetaStoreClient.invoke() should retry Trift exception even if not wrapped in MetaException

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-15937:
-


> RetryingMetaStoreClient.invoke() should retry Trift exception even if not 
> wrapped in MetaException
> --
>
> Key: HIVE-15937
> URL: https://issues.apache.org/jira/browse/HIVE-15937
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore, Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15936) ConcurrentModificationException in ATSHook

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-15936:
--
Attachment: HIVE-15936.2.patch

You are right. Miss the fact HookContext is constructed once and used in both 
pre-hook and post-hook. Attach a new patch.

> ConcurrentModificationException in ATSHook
> --
>
> Key: HIVE-15936
> URL: https://issues.apache.org/jira/browse/HIVE-15936
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15936.1.patch, HIVE-15936.2.patch
>
>
> See ATSHook error:
> {noformat}
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
> ~[?:1.8.0_112]
>   at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
> ~[?:1.8.0_112]
>   at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:290) 
> ~[guava-14.0.1.jar:?]
>   at 
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:219) 
> ~[hive-common-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> org.apache.hadoop.hive.ql.hooks.ATSHook.createPostHookEvent(ATSHook.java:347) 
> ~[hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:206) 
> [hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [?:1.8.0_112]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> According to [~jdere], ATSHook is currently accessing the PerfLogger on a 
> separate thread, which means the main query thread can potentially write to 
> the PerfLogger at the same time.
> The ATSHook should access the PerfLogger on the main query thread, before it 
> sends the execution to the ATS Logger thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15936) ConcurrentModificationException in ATSHook

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869011#comment-15869011
 ] 

Jason Dere commented on HIVE-15936:
---

I don't think the durations should be constructed at HookContext construction 
time - in Driver.execute() this happens at the start of query execution (and 
re-used for both pre-hook and post-hook). As a result there will be very few 
PerfLogger values at this time. I think a more appropriate place would be 
within the ATSHook, but before submitting the work to the ExecutorService.



> ConcurrentModificationException in ATSHook
> --
>
> Key: HIVE-15936
> URL: https://issues.apache.org/jira/browse/HIVE-15936
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15936.1.patch
>
>
> See ATSHook error:
> {noformat}
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
> ~[?:1.8.0_112]
>   at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
> ~[?:1.8.0_112]
>   at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:290) 
> ~[guava-14.0.1.jar:?]
>   at 
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:219) 
> ~[hive-common-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> org.apache.hadoop.hive.ql.hooks.ATSHook.createPostHookEvent(ATSHook.java:347) 
> ~[hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:206) 
> [hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [?:1.8.0_112]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> According to [~jdere], ATSHook is currently accessing the PerfLogger on a 
> separate thread, which means the main query thread can potentially write to 
> the PerfLogger at the same time.
> The ATSHook should access the PerfLogger on the main query thread, before it 
> sends the execution to the ATS Logger thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15910) Improvements in Hive Unit Test by using In-memory Derby DB

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869009#comment-15869009
 ] 

Hive QA commented on HIVE-15910:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852925/HIVE-15910.4.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10238 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31] 
(batchId=133)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorPartitionWithBase 
(batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorPartitionWithBaseMissingBuckets
 (batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableNoBase 
(batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableWithBase 
(batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorPartitionWithBase 
(batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorTableLegacy 
(batchId=246)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.minorTableNoBase 
(batchId=246)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3581/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3581/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3581/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852925 - PreCommit-HIVE-Build

> Improvements in Hive Unit Test by using In-memory Derby DB
> --
>
> Key: HIVE-15910
> URL: https://issues.apache.org/jira/browse/HIVE-15910
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Sankar Hariappan
>Assignee: Wei Zheng
> Attachments: HIVE-15910.01.patch, HIVE-15910.2.patch, 
> HIVE-15910.3.patch, HIVE-15910.4.patch
>
>
> Hive UT currently uses Derby DB with storage on disk which have some 
> practical problems.
> 1. The run-time of Hive unit tests are high as need to operate on the disk 
> quite often.
> 2. It can cause conflict if multiple test cases operates on the same table 
> name (such as table being created already exist).
> To solve these problems, we shall use an in-memory storage option of Derby DB 
> which can be even persisted if the test case demands that.
> https://db.apache.org/derby/docs/10.8/devguide/cdevdvlpinmemdb.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15936) ConcurrentModificationException in ATSHook

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-15936:
--
Status: Patch Available  (was: Open)

> ConcurrentModificationException in ATSHook
> --
>
> Key: HIVE-15936
> URL: https://issues.apache.org/jira/browse/HIVE-15936
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15936.1.patch
>
>
> See ATSHook error:
> {noformat}
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
> ~[?:1.8.0_112]
>   at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
> ~[?:1.8.0_112]
>   at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:290) 
> ~[guava-14.0.1.jar:?]
>   at 
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:219) 
> ~[hive-common-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> org.apache.hadoop.hive.ql.hooks.ATSHook.createPostHookEvent(ATSHook.java:347) 
> ~[hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:206) 
> [hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [?:1.8.0_112]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> According to [~jdere], ATSHook is currently accessing the PerfLogger on a 
> separate thread, which means the main query thread can potentially write to 
> the PerfLogger at the same time.
> The ATSHook should access the PerfLogger on the main query thread, before it 
> sends the execution to the ATS Logger thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15936) ConcurrentModificationException in ATSHook

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-15936:
--
Attachment: HIVE-15936.1.patch

> ConcurrentModificationException in ATSHook
> --
>
> Key: HIVE-15936
> URL: https://issues.apache.org/jira/browse/HIVE-15936
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15936.1.patch
>
>
> See ATSHook error:
> {noformat}
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
> ~[?:1.8.0_112]
>   at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
> ~[?:1.8.0_112]
>   at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:290) 
> ~[guava-14.0.1.jar:?]
>   at 
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:219) 
> ~[hive-common-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> org.apache.hadoop.hive.ql.hooks.ATSHook.createPostHookEvent(ATSHook.java:347) 
> ~[hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:206) 
> [hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [?:1.8.0_112]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> According to [~jdere], ATSHook is currently accessing the PerfLogger on a 
> separate thread, which means the main query thread can potentially write to 
> the PerfLogger at the same time.
> The ATSHook should access the PerfLogger on the main query thread, before it 
> sends the execution to the ATS Logger thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15936) ConcurrentModificationException in ATSHook

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned HIVE-15936:
-


> ConcurrentModificationException in ATSHook
> --
>
> Key: HIVE-15936
> URL: https://issues.apache.org/jira/browse/HIVE-15936
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15936.1.patch
>
>
> See ATSHook error:
> {noformat}
> java.util.ConcurrentModificationException
>   at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
> ~[?:1.8.0_112]
>   at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
> ~[?:1.8.0_112]
>   at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
> ~[?:1.8.0_112]
>   at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:290) 
> ~[guava-14.0.1.jar:?]
>   at 
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:219) 
> ~[hive-common-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> org.apache.hadoop.hive.ql.hooks.ATSHook.createPostHookEvent(ATSHook.java:347) 
> ~[hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:206) 
> [hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [?:1.8.0_112]
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [?:1.8.0_112]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [?:1.8.0_112]
>   at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
> According to [~jdere], ATSHook is currently accessing the PerfLogger on a 
> separate thread, which means the main query thread can potentially write to 
> the PerfLogger at the same time.
> The ATSHook should access the PerfLogger on the main query thread, before it 
> sends the execution to the ATS Logger thread.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15902) Select query involving date throwing Hive 2 Internal error: unsupported conversion from type: date

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868975#comment-15868975
 ] 

Matt McCline commented on HIVE-15902:
-

+1 LGTM

> Select query involving date throwing Hive 2 Internal error: unsupported 
> conversion from type: date
> --
>
> Key: HIVE-15902
> URL: https://issues.apache.org/jira/browse/HIVE-15902
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Jason Dere
> Attachments: HIVE-15902.1.patch
>
>
> The following query is throwing Hive 2 Internal error: unsupported conversion 
> from type: date
> Query:
> create table table_one (ts timestamp, dt date) stored as orc;
> insert into table_one values ('2034-08-04 17:42:59','2038-07-01');
> insert into table_one values ('2031-02-07 13:02:38','2072-10-19');
> create table table_two (ts timestamp, dt date) stored as orc;
> insert into table_two values ('2069-04-01 09:05:54','1990-10-12');
> insert into table_two values ('2031-02-07 13:02:38','2072-10-19');
> create table table_three as
> select count(*) from table_one
> group by ts,dt
> having dt in (select dt from table_two);
> Error while running task ( failure ) : 
> attempt_1486991777989_0184_18_02_00_0:java.lang.RuntimeException: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1833)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
>   at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
>   at 
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:420)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
>   ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
>   ... 18 more
> Caused by: java.lang.RuntimeException: Hive 2 Internal error: unsupported 
> conversion from type: date
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getLong(PrimitiveObjectInspectorUtils.java:770)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterLongColumnBetweenDynamicValue.evaluate(FilterLongColumnBetweenDynamicValue.java:82)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:39)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:112)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:883)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:783)
>   ... 19 more



--

[jira] [Updated] (HIVE-15935) ACL is not set in ATS data

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-15935:
--
Attachment: HIVE-15935.1.patch

> ACL is not set in ATS data
> --
>
> Key: HIVE-15935
> URL: https://issues.apache.org/jira/browse/HIVE-15935
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15935.1.patch
>
>
> When publishing ATS info, Hive does not set ACL, that make Hive ATS entries 
> visible to all users. On the other hand, Tez ATS entires is using Tez DAG ACL 
> which limit both view/modify ACL to end user only. We shall make them 
> consistent. In the Jira, I am going to limit ACL to end user for both Tez ATS 
> and Hive ATS, also provide config "hive.view.acls" and "hive.modify.acls" if 
> user need to overridden.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15935) ACL is not set in ATS data

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated HIVE-15935:
--
Status: Patch Available  (was: Open)

> ACL is not set in ATS data
> --
>
> Key: HIVE-15935
> URL: https://issues.apache.org/jira/browse/HIVE-15935
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
> Attachments: HIVE-15935.1.patch
>
>
> When publishing ATS info, Hive does not set ACL, that make Hive ATS entries 
> visible to all users. On the other hand, Tez ATS entires is using Tez DAG ACL 
> which limit both view/modify ACL to end user only. We shall make them 
> consistent. In the Jira, I am going to limit ACL to end user for both Tez ATS 
> and Hive ATS, also provide config "hive.view.acls" and "hive.modify.acls" if 
> user need to overridden.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15902) Select query involving date throwing Hive 2 Internal error: unsupported conversion from type: date

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868964#comment-15868964
 ] 

Jason Dere commented on HIVE-15902:
---

Test results look good. [~mmccline] can you review?

> Select query involving date throwing Hive 2 Internal error: unsupported 
> conversion from type: date
> --
>
> Key: HIVE-15902
> URL: https://issues.apache.org/jira/browse/HIVE-15902
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 2.1.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Jason Dere
> Attachments: HIVE-15902.1.patch
>
>
> The following query is throwing Hive 2 Internal error: unsupported conversion 
> from type: date
> Query:
> create table table_one (ts timestamp, dt date) stored as orc;
> insert into table_one values ('2034-08-04 17:42:59','2038-07-01');
> insert into table_one values ('2031-02-07 13:02:38','2072-10-19');
> create table table_two (ts timestamp, dt date) stored as orc;
> insert into table_two values ('2069-04-01 09:05:54','1990-10-12');
> insert into table_two values ('2031-02-07 13:02:38','2072-10-19');
> create table table_three as
> select count(*) from table_one
> group by ts,dt
> having dt in (select dt from table_two);
> Error while running task ( failure ) : 
> attempt_1486991777989_0184_18_02_00_0:java.lang.RuntimeException: 
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
> Hive Runtime Error while processing row
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1833)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
>   at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
>   at 
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while 
> processing row
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:420)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
>   ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime 
> Error while processing row
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
>   ... 18 more
> Caused by: java.lang.RuntimeException: Hive 2 Internal error: unsupported 
> conversion from type: date
>   at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getLong(PrimitiveObjectInspectorUtils.java:770)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterLongColumnBetweenDynamicValue.evaluate(FilterLongColumnBetweenDynamicValue.java:82)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.expressions.FilterExprAndExpr.evaluate(FilterExprAndExpr.java:39)
>   at 
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:112)
>   at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:883)
>   at 
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
>   at 
> 

[jira] [Commented] (HIVE-15923) Hive default partition causes errors in get partitions

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868955#comment-15868955
 ] 

Hive QA commented on HIVE-15923:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852913/HIVE-15923.01.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3580/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3580/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3580/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-02-16 01:41:09.942
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-3580/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-02-16 01:41:09.945
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b63f7d7 HIVE-15892: Vectorization: Fast Hash tables need to do 
bounds checking during expand (Matt McCline, reviewed by Jason Dere
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b63f7d7 HIVE-15892: Vectorization: Fast Hash tables need to do 
bounds checking during expand (Matt McCline, reviewed by Jason Dere
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-02-16 01:41:11.028
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: patch -p0
patching file 
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantDefaultEvaluator.java
patching file 
ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java
patching file ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
patching file 
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ppr/PartitionPruner.java
patching file 
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
patching file 
ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java
patching file 
ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeConstantDefaultDesc.java
patching file 
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFIsDefaultPartition.java
patching file 
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPEqual.java
patching file 
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFOPNotEqual.java
patching file ql/src/test/queries/clientnegative/drop_default_partition_filter.q
patching file ql/src/test/queries/clientpositive/drop_partitions_filter4.q
patching file ql/src/test/queries/clientpositive/partitions_filter_default.q
patching file 
ql/src/test/results/clientnegative/drop_default_partition_filter.q.out
patching file ql/src/test/results/clientpositive/drop_partitions_filter4.q.out
patching file ql/src/test/results/clientpositive/partitions_filter_default.q.out
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
ANTLR Parser Generator  Version 3.5.2
Output file 
/data/hiveptest/working/apache-github-source-source/metastore/target/generated-sources/antlr3/org/apache/hadoop/hive/metastore/parser/FilterParser.java
 does not exist: must build 
/data/hiveptest/working/apache-github-source-source/metastore/src/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
DataNucleus Enhancer (version 4.1.6) for API "JDO"
DataNucleus Enhancer : Classpath
>>  /usr/share/maven/boot/plexus-classworlds-2.x.jar
ENHANCED (Persistable) : 

[jira] [Commented] (HIVE-15877) Upload dependency jars for druid storage handler

[Ashutosh Chauhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868953#comment-15868953
 ] 

Ashutosh Chauhan commented on HIVE-15877:
-

* Instead of apache commons logger, use slf4j Logger.
* Much of this code already exists in Utilities::jarFinderGetJar() which is 
called by LlapServiceDriver::localizeJarForClass() for same purposes. Perhaps, 
you can reuse that instead of repeated logic here.

> Upload dependency jars for druid storage handler
> 
>
> Key: HIVE-15877
> URL: https://issues.apache.org/jira/browse/HIVE-15877
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: slim bouguerra
>Assignee: slim bouguerra
> Attachments: HIVE-15877.patch
>
>
> Upload dependency jars for druid storage handler



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15929) Fix HiveDecimalWritable to be compatible with Hive 2.1

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868951#comment-15868951
 ] 

Hive QA commented on HIVE-15929:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852911/HIVE-15929.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10238 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31] 
(batchId=133)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3579/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3579/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3579/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852911 - PreCommit-HIVE-Build

> Fix HiveDecimalWritable to be compatible with Hive 2.1
> --
>
> Key: HIVE-15929
> URL: https://issues.apache.org/jira/browse/HIVE-15929
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-15929.patch
>
>
> HIVE-15335 broke compatibility with Hive 2.1 by making 
> HiveDecimalWritable.getInternalStorate() throw an exception when called on an 
> unset value. It is easy to instead return an empty array, which will allow 
> the old code to allocate a new array.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15919) Row count mismatch for count * query

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-15919:
--
Status: Patch Available  (was: Open)

> Row count mismatch for count * query
> 
>
> Key: HIVE-15919
> URL: https://issues.apache.org/jira/browse/HIVE-15919
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Jason Dere
> Attachments: HIVE-15919.1.patch, table_14.q, table_6.q
>
>
> The following query is returning different output when run against hive and 
> postgres.
> Query:
> SELECT COUNT (*)
> FROM
> (SELECT LAG(COALESCE(t2.int_col_14, t1.int_col_80),22) OVER (ORDER BY 
> t1.tinyint_col_52 DESC) AS int_col
> FROM table_6 t1
> INNER JOIN table_14 t2 ON ((t2.decimal0101_col_55) = (t1.decimal0101_col_9))) 
> AS FOO;
> From hive: 0
> From postgres: 66903279
> Attaching ddl files for the tables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15919) Row count mismatch for count * query

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-15919:
--
Attachment: HIVE-15919.1.patch

Patch to try to restrict the automatic conversion behavior to just numeric 
functions.

> Row count mismatch for count * query
> 
>
> Key: HIVE-15919
> URL: https://issues.apache.org/jira/browse/HIVE-15919
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Jason Dere
> Attachments: HIVE-15919.1.patch, table_14.q, table_6.q
>
>
> The following query is returning different output when run against hive and 
> postgres.
> Query:
> SELECT COUNT (*)
> FROM
> (SELECT LAG(COALESCE(t2.int_col_14, t1.int_col_80),22) OVER (ORDER BY 
> t1.tinyint_col_52 DESC) AS int_col
> FROM table_6 t1
> INNER JOIN table_14 t2 ON ((t2.decimal0101_col_55) = (t1.decimal0101_col_9))) 
> AS FOO;
> From hive: 0
> From postgres: 66903279
> Attaching ddl files for the tables.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15935) ACL is not set in ATS data

[Daniel Dai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai reassigned HIVE-15935:
-


> ACL is not set in ATS data
> --
>
> Key: HIVE-15935
> URL: https://issues.apache.org/jira/browse/HIVE-15935
> Project: Hive
>  Issue Type: Bug
>Reporter: Daniel Dai
>Assignee: Daniel Dai
>
> When publishing ATS info, Hive does not set ACL, that make Hive ATS entries 
> visible to all users. On the other hand, Tez ATS entires is using Tez DAG ACL 
> which limit both view/modify ACL to end user only. We shall make them 
> consistent. In the Jira, I am going to limit ACL to end user for both Tez ATS 
> and Hive ATS, also provide config "hive.view.acls" and "hive.modify.acls" if 
> user need to overridden.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15891) Detect query rewrite scenario for UPDATE/DELETE/MERGE and fail fast

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868933#comment-15868933
 ] 

Eugene Koifman commented on HIVE-15891:
---

does it fail any Merge or some specific one?

> Detect query rewrite scenario for UPDATE/DELETE/MERGE and fail fast
> ---
>
> Key: HIVE-15891
> URL: https://issues.apache.org/jira/browse/HIVE-15891
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15891.1.patch, HIVE-15891.2.patch
>
>
> Currently ACID UpdateDeleteSemanticAnalyzer directly manipulates the AST tree 
> but it's different from the general approach of modifying the token stream 
> and thus will cause AST tree mismatch if there is any rewrite happening after 
> UpdateDeleteSemanticAnalyzer.
> The long term solution will be to rewrite the AST handling logic in 
> UpdateDeleteSemanticAnalyzer, to make it consistent with the general approach.
> This ticket will for now detect the error prone cases and fail early. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15891) Detect query rewrite scenario for UPDATE/DELETE/MERGE and fail fast

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868906#comment-15868906
 ] 

Wei Zheng commented on HIVE-15891:
--

I tried that, and it works for existing tests that contain subquery. But this 
check will fail MERGE statement.
{code}
if (!hasSubQuery) {
  throw new RuntimeException("A matching node should have been found, 
otherwise it means " +
  "this is an UPDATE/DELETE/MERGE query but the AST got modified!");
}
{code}
Do we want to add another condition to rule out MERGE?

> Detect query rewrite scenario for UPDATE/DELETE/MERGE and fail fast
> ---
>
> Key: HIVE-15891
> URL: https://issues.apache.org/jira/browse/HIVE-15891
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15891.1.patch, HIVE-15891.2.patch
>
>
> Currently ACID UpdateDeleteSemanticAnalyzer directly manipulates the AST tree 
> but it's different from the general approach of modifying the token stream 
> and thus will cause AST tree mismatch if there is any rewrite happening after 
> UpdateDeleteSemanticAnalyzer.
> The long term solution will be to rewrite the AST handling logic in 
> UpdateDeleteSemanticAnalyzer, to make it consistent with the general approach.
> This ticket will for now detect the error prone cases and fail early. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15708) Upgrade calcite version to 1.11

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868902#comment-15868902
 ] 

Hive QA commented on HIVE-15708:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852906/HIVE-15708.09.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 62 failed/errored test(s), 10238 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ambiguitycheck] 
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_select] 
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_timestamp] 
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cast1] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_outer_join_ppr] 
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_udaf_percentile_approx_23]
 (batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constantfolding] 
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog2] (batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_2] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_intervals] 
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn] (batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_cond_pushdown] 
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fouter_join_ppr] 
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto_unused] 
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_merging] 
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[louter_join_ppr] 
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite]
 (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_rewrite_multi_db]
 (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoins] (batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ops_comparison] 
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[outer_join_ppr] 
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_decimal] 
(batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_outer_join1] 
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[router_join_ppr] 
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamp] (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[type_conversions_1] 
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_percentile_approx_23]
 (batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf3] (batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_reflect2] 
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_interval_mapjoin] 
(batchId=35)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_smb_mapjoin_14]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_9]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
 (batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1]
 (batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3] 
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_decimal]
 (batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_auto_smb_mapjoin_14]
 (batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_2]
 (batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_interval_mapjoin]
 (batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_0]
 (batchId=154)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
 (batchId=161)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)

[jira] [Updated] (HIVE-15933) Improve plans for correlated subquery with join and predicate

[Ashutosh Chauhan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Chauhan updated HIVE-15933:

Status: Patch Available  (was: Open)

> Improve plans for correlated subquery with join and predicate
> -
>
> Key: HIVE-15933
> URL: https://issues.apache.org/jira/browse/HIVE-15933
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-15933.1.patch
>
>
> This is a continuation of HIVE-15905
> for queries such as:
> {code:SQL}
> explain select  
>   cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   count(*) cnt1,
>   cd_purchase_estimate,
>   count(*) cnt2,
>   cd_credit_rating,
>   count(*) cnt3,
>   cd_dep_count,
>   count(*) cnt4,
>   cd_dep_employed_count,
>   count(*) cnt5,
>   cd_dep_college_count,
>   count(*) cnt6
>  from
>   customer c,customer_address ca,customer_demographics
>  where
>   c.c_current_addr_sk = ca.ca_address_sk and
>   ca_county in ('Walker County','Richland County','Gaines County','Douglas 
> County','Dona Ana County') and
>   cd_demo_sk = c.c_current_cdemo_sk and 
>   exists (select *
>   from store_sales,date_dim
>   where c.c_customer_sk = ss_customer_sk and
> ss_sold_date_sk = d_date_sk and
> d_year = 2002 and
> d_moy between 4 and 4+3)
>  group by cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   cd_purchase_estimate,
>   cd_credit_rating,
>   cd_dep_count,
>   cd_dep_employed_count,
>   cd_dep_college_count
>  order by cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   cd_purchase_estimate,
>   cd_credit_rating,
>   cd_dep_count,
>   cd_dep_employed_count,
>   cd_dep_college_count
> limit 100;
> {code}
> HIVE generates un-necessary joins to produce value for correlated columns.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15841) Upgrade Hive to ORC 1.3.2

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868894#comment-15868894
 ] 

Owen O'Malley commented on HIVE-15841:
--

It gets them from the SchemaEvolution. In particular, they come from 
OrcInputFormat.getDesiredRowTypeDescr(...) which uses schema.evolution.columns 
and schema.evolution.columns.types or columns and columns.types from the config.

> Upgrade Hive to ORC 1.3.2
> -
>
> Key: HIVE-15841
> URL: https://issues.apache.org/jira/browse/HIVE-15841
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-15841.patch
>
>
> Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.2 once it 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15934) Downgrade Maven surefire plugin from 2.19.1 to 2.18.1

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15934:
-
Status: Patch Available  (was: Open)

> Downgrade Maven surefire plugin from 2.19.1 to 2.18.1
> -
>
> Key: HIVE-15934
> URL: https://issues.apache.org/jira/browse/HIVE-15934
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15934.1.patch
>
>
> Surefire 2.19.1 has some issue 
> (https://issues.apache.org/jira/browse/SUREFIRE-1255) which caused debugging 
> session to abort after a short period of time. Many IntelliJ users have seen 
> this, although it looks fine for Eclipse users. Version 2.18.1 works fine.
> We'd better make the change to not impact the development for IntelliJ guys. 
> We can upgrade again once the root cause is figured out.
> cc [~kgyrtkirk] [~ashutoshc]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15934) Downgrade Maven surefire plugin from 2.19.1 to 2.18.1

[Wei Zheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868886#comment-15868886
 ] 

Wei Zheng commented on HIVE-15934:
--

[~ashutoshc] Can you take a look please?

> Downgrade Maven surefire plugin from 2.19.1 to 2.18.1
> -
>
> Key: HIVE-15934
> URL: https://issues.apache.org/jira/browse/HIVE-15934
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15934.1.patch
>
>
> Surefire 2.19.1 has some issue 
> (https://issues.apache.org/jira/browse/SUREFIRE-1255) which caused debugging 
> session to abort after a short period of time. Many IntelliJ users have seen 
> this, although it looks fine for Eclipse users. Version 2.18.1 works fine.
> We'd better make the change to not impact the development for IntelliJ guys. 
> We can upgrade again once the root cause is figured out.
> cc [~kgyrtkirk] [~ashutoshc]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15934) Downgrade Maven surefire plugin from 2.19.1 to 2.18.1

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15934:
-
Attachment: HIVE-15934.1.patch

> Downgrade Maven surefire plugin from 2.19.1 to 2.18.1
> -
>
> Key: HIVE-15934
> URL: https://issues.apache.org/jira/browse/HIVE-15934
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15934.1.patch
>
>
> Surefire 2.19.1 has some issue 
> (https://issues.apache.org/jira/browse/SUREFIRE-1255) which caused debugging 
> session to abort after a short period of time. Many IntelliJ users have seen 
> this, although it looks fine for Eclipse users. Version 2.18.1 works fine.
> We'd better make the change to not impact the development for IntelliJ guys. 
> We can upgrade again once the root cause is figured out.
> cc [~kgyrtkirk] [~ashutoshc]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15934) Downgrade Maven surefire plugin from 2.19.1 to 2.18.1

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15934:
-
Component/s: Tests

> Downgrade Maven surefire plugin from 2.19.1 to 2.18.1
> -
>
> Key: HIVE-15934
> URL: https://issues.apache.org/jira/browse/HIVE-15934
> Project: Hive
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> Surefire 2.19.1 has some issue 
> (https://issues.apache.org/jira/browse/SUREFIRE-1255) which caused debugging 
> session to abort after a short period of time. Many IntelliJ users have seen 
> this, although it looks fine for Eclipse users. Version 2.18.1 works fine.
> We'd better make the change to not impact the development for IntelliJ guys. 
> We can upgrade again once the root cause is figured out.
> cc [~kgyrtkirk] [~ashutoshc]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15934) Downgrade Maven surefire plugin from 2.19.1 to 2.18.1

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng reassigned HIVE-15934:



> Downgrade Maven surefire plugin from 2.19.1 to 2.18.1
> -
>
> Key: HIVE-15934
> URL: https://issues.apache.org/jira/browse/HIVE-15934
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> Surefire 2.19.1 has some issue 
> (https://issues.apache.org/jira/browse/SUREFIRE-1255) which caused debugging 
> session to abort after a short period of time. Many IntelliJ users have seen 
> this, although it looks fine for Eclipse users. Version 2.18.1 works fine.
> We'd better make the change to not impact the development for IntelliJ guys. 
> We can upgrade again once the root cause is figured out.
> cc [~kgyrtkirk] [~ashutoshc]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (HIVE-15892) Vectorization: Fast Hash tables need to do bounds checking during expand

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline resolved HIVE-15892.
-
Resolution: Fixed

> Vectorization: Fast Hash tables need to do bounds checking during expand
> 
>
> Key: HIVE-15892
> URL: https://issues.apache.org/jira/browse/HIVE-15892
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15892.01.patch, HIVE-15892.02.patch
>
>
> VectorMapJoinFastLongHashTable line 165 gets NegativeArraySizeException:
> {code}
> long[] newSlotPairs = new long[newSlotPairArraySize];
> {code}
> We need to add a size check... Java math for this wrapped around to negative:
> {code}
> int newSlotPairArraySize = newLogicalHashBucketCount * 2;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15892) Vectorization: Fast Hash tables need to do bounds checking during expand

[Matt McCline (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt McCline updated HIVE-15892:

Fix Version/s: 2.2.0

> Vectorization: Fast Hash tables need to do bounds checking during expand
> 
>
> Key: HIVE-15892
> URL: https://issues.apache.org/jira/browse/HIVE-15892
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-15892.01.patch, HIVE-15892.02.patch
>
>
> VectorMapJoinFastLongHashTable line 165 gets NegativeArraySizeException:
> {code}
> long[] newSlotPairs = new long[newSlotPairArraySize];
> {code}
> We need to add a size check... Java math for this wrapped around to negative:
> {code}
> int newSlotPairArraySize = newLogicalHashBucketCount * 2;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15892) Vectorization: Fast Hash tables need to do bounds checking during expand

[Matt McCline (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868871#comment-15868871
 ] 

Matt McCline commented on HIVE-15892:
-

Thanks [~jdere] and [~gopalv] for your help.
Committed to master.

> Vectorization: Fast Hash tables need to do bounds checking during expand
> 
>
> Key: HIVE-15892
> URL: https://issues.apache.org/jira/browse/HIVE-15892
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-15892.01.patch, HIVE-15892.02.patch
>
>
> VectorMapJoinFastLongHashTable line 165 gets NegativeArraySizeException:
> {code}
> long[] newSlotPairs = new long[newSlotPairArraySize];
> {code}
> We need to add a size check... Java math for this wrapped around to negative:
> {code}
> int newSlotPairArraySize = newLogicalHashBucketCount * 2;
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15933) Improve plans for correlated subquery with join and predicate

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-15933:
---
Attachment: HIVE-15933.1.patch

> Improve plans for correlated subquery with join and predicate
> -
>
> Key: HIVE-15933
> URL: https://issues.apache.org/jira/browse/HIVE-15933
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-15933.1.patch
>
>
> This is a continuation of HIVE-15905
> for queries such as:
> {code:SQL}
> explain select  
>   cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   count(*) cnt1,
>   cd_purchase_estimate,
>   count(*) cnt2,
>   cd_credit_rating,
>   count(*) cnt3,
>   cd_dep_count,
>   count(*) cnt4,
>   cd_dep_employed_count,
>   count(*) cnt5,
>   cd_dep_college_count,
>   count(*) cnt6
>  from
>   customer c,customer_address ca,customer_demographics
>  where
>   c.c_current_addr_sk = ca.ca_address_sk and
>   ca_county in ('Walker County','Richland County','Gaines County','Douglas 
> County','Dona Ana County') and
>   cd_demo_sk = c.c_current_cdemo_sk and 
>   exists (select *
>   from store_sales,date_dim
>   where c.c_customer_sk = ss_customer_sk and
> ss_sold_date_sk = d_date_sk and
> d_year = 2002 and
> d_moy between 4 and 4+3)
>  group by cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   cd_purchase_estimate,
>   cd_credit_rating,
>   cd_dep_count,
>   cd_dep_employed_count,
>   cd_dep_college_count
>  order by cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   cd_purchase_estimate,
>   cd_credit_rating,
>   cd_dep_count,
>   cd_dep_employed_count,
>   cd_dep_college_count
> limit 100;
> {code}
> HIVE generates un-necessary joins to produce value for correlated columns.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-6590) Hive does not work properly with boolean partition columns (wrong results and inserts to incorrect HDFS path)

[Alan Gates (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-6590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868867#comment-15868867
 ] 

Alan Gates commented on HIVE-6590:
--

I agree with Jason that we don't want to change 
PrimitiveObjectInspectorUtils.getBoolean().  That's just too fundamental of a 
change and risks breaking unknown things.  I also agree that it is better to 
see if we can special case this in the partitioning code to do the right thing 
for partitions.

> Hive does not work properly with boolean partition columns (wrong results and 
> inserts to incorrect HDFS path)
> -
>
> Key: HIVE-6590
> URL: https://issues.apache.org/jira/browse/HIVE-6590
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema, Metastore
>Affects Versions: 0.10.0
>Reporter: Lenni Kuff
>Assignee: Zoltan Haindrich
> Attachments: HIVE-6590.1.patch, HIVE-6590.2.patch, HIVE-6590.3.patch
>
>
> Hive does not work properly with boolean partition columns. Queries return 
> wrong results and also insert to incorrect HDFS paths.
> {code}
> create table bool_part(int_col int) partitioned by(bool_col boolean);
> # This works, creating 3 unique partitions!
> ALTER TABLE bool_table ADD PARTITION (bool_col=FALSE);
> ALTER TABLE bool_table ADD PARTITION (bool_col=false);
> ALTER TABLE bool_table ADD PARTITION (bool_col=False);
> {code}
> The first problem is that Hive cannot filter on a bool partition key column. 
> "select * from bool_part" returns the correct results, but if you apply a 
> filter on the bool partition key column hive won't return any results.
> The second problem is that Hive seems to just call "toString()" on the 
> boolean literal value. This means you can end up with multiple partitions 
> (FALSE, false, FaLSE, etc) mapping to the literal value 'FALSE'. For example, 
> if you can add three partition in have for the same logic value "false" doing:
> ALTER TABLE bool_table ADD PARTITION (bool_col=FALSE) -> 
> /test-warehouse/bool_table/bool_col=FALSE/
> ALTER TABLE bool_table ADD PARTITION (bool_col=false) -> 
> /test-warehouse/bool_table/bool_col=false/
> ALTER TABLE bool_table ADD PARTITION (bool_col=False) -> 
> /test-warehouse/bool_table/bool_col=False/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15917) incorrect error handling from BackgroundWork can cause beeline query to hang

[Siddharth Seth (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868852#comment-15868852
 ] 

Siddharth Seth commented on HIVE-15917:
---

+1. Discussed offline, this is while a query is running.

> incorrect error handling from BackgroundWork can cause beeline query to hang
> 
>
> Key: HIVE-15917
> URL: https://issues.apache.org/jira/browse/HIVE-15917
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-15917.01.patch, HIVE-15917.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15900) Beeline prints tez job progress in stdout instead of stderr

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868833#comment-15868833
 ] 

Hive QA commented on HIVE-15900:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852894/HIVE-15900.3.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 10224 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
 (batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_sortmerge_join_4]
 (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby10] 
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby3_noskew_multi_distinct]
 (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_multi_single_reducer]
 (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31] 
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_partition_metadataonly]
 (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[load_dyn_part4] 
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[mapjoin_filter_on_outerjoin]
 (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_14] 
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats18] 
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union3] 
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_21] 
(batchId=121)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3577/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3577/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3577/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852894 - PreCommit-HIVE-Build

> Beeline prints tez job progress in stdout instead of stderr
> ---
>
> Key: HIVE-15900
> URL: https://issues.apache.org/jira/browse/HIVE-15900
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 2.2.0
>Reporter: Aswathy Chellammal Sreekumar
>Assignee: Thejas M Nair
> Attachments: HIVE-15900.1.patch, HIVE-15900.2.patch, 
> HIVE-15900.3.patch, std_out
>
>
> Tez job progress messages are getting updated to stdout instead of stderr
> Attaching output file for below command, with the tez job status printed
> $HIVE_HOME/bin/beeline -n  -p  -u " --outputformat=tsv -e "analyze table studenttab10k compute statistics;" > 
> stdout



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15923) Hive default partition causes errors in get partitions

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868832#comment-15868832
 ] 

Aihua Xu commented on HIVE-15923:
-

I would be glad if we can remove the change from the udf.

>From the change, can you check drop partition with condition like {{x != 3}} 
>works if  there is a default partition? I guess, the default one won't get 
>dropped.

> Hive default partition causes errors in get partitions
> --
>
> Key: HIVE-15923
> URL: https://issues.apache.org/jira/browse/HIVE-15923
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15923.01.patch, HIVE-15923.patch
>
>
> This is the ORM error, direct SQL fails too before that, with a similar error.
> {noformat}
> 2017-02-14T17:45:11,158 ERROR [09fdd887-0164-4f55-97e9-4ba147d962be main] 
> metastore.ObjectStore:java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDefaultDesc cannot be cast to 
> java.lang.Long
> at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaLongObjectInspector.get(JavaLongObjectInspector.java:40)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getDouble(PrimitiveObjectInspectorUtils.java:801)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DoubleConverter.convert(P
> rimitiveObjectInspectorConverter.java:240) 
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqualOrGreaterThan.evaluate(GenericUDFOPEqualOrGreaterThan.java:145)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween.evaluate(GenericUDFBetween.java:57)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:88)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:63)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.evaluateExprOnPart(PartExprEvalUtils.java:126)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15841) Upgrade Hive to ORC 1.3.2

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868830#comment-15868830
 ] 

Sergey Shelukhin commented on HIVE-15841:
-

How does sarg applier know the column names now?

> Upgrade Hive to ORC 1.3.2
> -
>
> Key: HIVE-15841
> URL: https://issues.apache.org/jira/browse/HIVE-15841
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-15841.patch
>
>
> Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.2 once it 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15931) JDBC: Improve logging when using ZooKeeper

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-15931:

Status: Patch Available  (was: Open)

> JDBC: Improve logging when using ZooKeeper
> --
>
> Key: HIVE-15931
> URL: https://issues.apache.org/jira/browse/HIVE-15931
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15931.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15931) JDBC: Improve logging when using ZooKeeper

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-15931:

Attachment: HIVE-15931.1.patch

> JDBC: Improve logging when using ZooKeeper
> --
>
> Key: HIVE-15931
> URL: https://issues.apache.org/jira/browse/HIVE-15931
> Project: Hive
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
> Attachments: HIVE-15931.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15841) Upgrade Hive to ORC 1.3.2

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-15841:
-
Attachment: HIVE-15841.patch

> Upgrade Hive to ORC 1.3.2
> -
>
> Key: HIVE-15841
> URL: https://issues.apache.org/jira/browse/HIVE-15841
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-15841.patch
>
>
> Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.2 once it 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15841) Upgrade Hive to ORC 1.3.2

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley reassigned HIVE-15841:


Assignee: Owen O'Malley

> Upgrade Hive to ORC 1.3.2
> -
>
> Key: HIVE-15841
> URL: https://issues.apache.org/jira/browse/HIVE-15841
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-15841.patch
>
>
> Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.2 once it 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15841) Upgrade Hive to ORC 1.3.2

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-15841:
-
Status: Patch Available  (was: Open)

> Upgrade Hive to ORC 1.3.2
> -
>
> Key: HIVE-15841
> URL: https://issues.apache.org/jira/browse/HIVE-15841
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
> Attachments: HIVE-15841.patch
>
>
> Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.2 once it 
> releases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (HIVE-15933) Improve plans for correlated subquery with join and predicate

[Vineet Garg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg reassigned HIVE-15933:
--


> Improve plans for correlated subquery with join and predicate
> -
>
> Key: HIVE-15933
> URL: https://issues.apache.org/jira/browse/HIVE-15933
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>
> This is a continuation of HIVE-15905
> for queries such as:
> {code:SQL}
> explain select  
>   cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   count(*) cnt1,
>   cd_purchase_estimate,
>   count(*) cnt2,
>   cd_credit_rating,
>   count(*) cnt3,
>   cd_dep_count,
>   count(*) cnt4,
>   cd_dep_employed_count,
>   count(*) cnt5,
>   cd_dep_college_count,
>   count(*) cnt6
>  from
>   customer c,customer_address ca,customer_demographics
>  where
>   c.c_current_addr_sk = ca.ca_address_sk and
>   ca_county in ('Walker County','Richland County','Gaines County','Douglas 
> County','Dona Ana County') and
>   cd_demo_sk = c.c_current_cdemo_sk and 
>   exists (select *
>   from store_sales,date_dim
>   where c.c_customer_sk = ss_customer_sk and
> ss_sold_date_sk = d_date_sk and
> d_year = 2002 and
> d_moy between 4 and 4+3)
>  group by cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   cd_purchase_estimate,
>   cd_credit_rating,
>   cd_dep_count,
>   cd_dep_employed_count,
>   cd_dep_college_count
>  order by cd_gender,
>   cd_marital_status,
>   cd_education_status,
>   cd_purchase_estimate,
>   cd_credit_rating,
>   cd_dep_count,
>   cd_dep_employed_count,
>   cd_dep_college_count
> limit 100;
> {code}
> HIVE generates un-necessary joins to produce value for correlated columns.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15930) Syntax error in the MySQL metastore schema definition

[Deepesh Khandelwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868780#comment-15868780
 ] 

Deepesh Khandelwal commented on HIVE-15930:
---

Thanks [~sershe] for the review and commit!

> Syntax error in the MySQL metastore schema definition
> -
>
> Key: HIVE-15930
> URL: https://issues.apache.org/jira/browse/HIVE-15930
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Reporter: Deepesh Khandelwal
>Assignee: Deepesh Khandelwal
>Priority: Critical
> Fix For: 2.2.0
>
> Attachments: HIVE-15930.patch
>
>
> The MySQL metastore definition 
> metastore/scripts/upgrade/mysql/hive-schema-2.2.0.mysql.sql contains a syntax 
> error for the table NOTIFICATION_LOG:
> {noformat}
> CREATE TABLE IF NOT EXISTS `NOTIFICATION_LOG`
> (
> `NL_ID` BIGINT(20) NOT NULL,
> `EVENT_ID` BIGINT(20) NOT NULL,
> `EVENT_TIME` INT(11) NOT NULL,
> `EVENT_TYPE` varchar(32) NOT NULL,
> `DB_NAME` varchar(128),
> `TBL_NAME` varchar(128),
> `MESSAGE` longtext,
> `MESSAGE_FORMAT` varchar(16)
> PRIMARY KEY (`NL_ID`)
> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> {noformat}
> Notice a comma is missing after {{`MESSAGE_FORMAT` varchar(16)}}.
> This got introduced through HIVE-10562.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15910) Improvements in Hive Unit Test by using In-memory Derby DB

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15910:
-
Attachment: (was: HIVE-15910.4.patch)

> Improvements in Hive Unit Test by using In-memory Derby DB
> --
>
> Key: HIVE-15910
> URL: https://issues.apache.org/jira/browse/HIVE-15910
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Sankar Hariappan
>Assignee: Wei Zheng
> Attachments: HIVE-15910.01.patch, HIVE-15910.2.patch, 
> HIVE-15910.3.patch, HIVE-15910.4.patch
>
>
> Hive UT currently uses Derby DB with storage on disk which have some 
> practical problems.
> 1. The run-time of Hive unit tests are high as need to operate on the disk 
> quite often.
> 2. It can cause conflict if multiple test cases operates on the same table 
> name (such as table being created already exist).
> To solve these problems, we shall use an in-memory storage option of Derby DB 
> which can be even persisted if the test case demands that.
> https://db.apache.org/derby/docs/10.8/devguide/cdevdvlpinmemdb.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15910) Improvements in Hive Unit Test by using In-memory Derby DB

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15910:
-
Attachment: HIVE-15910.4.patch

patch 4 for debugging

> Improvements in Hive Unit Test by using In-memory Derby DB
> --
>
> Key: HIVE-15910
> URL: https://issues.apache.org/jira/browse/HIVE-15910
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Sankar Hariappan
>Assignee: Wei Zheng
> Attachments: HIVE-15910.01.patch, HIVE-15910.2.patch, 
> HIVE-15910.3.patch, HIVE-15910.4.patch
>
>
> Hive UT currently uses Derby DB with storage on disk which have some 
> practical problems.
> 1. The run-time of Hive unit tests are high as need to operate on the disk 
> quite often.
> 2. It can cause conflict if multiple test cases operates on the same table 
> name (such as table being created already exist).
> To solve these problems, we shall use an in-memory storage option of Derby DB 
> which can be even persisted if the test case demands that.
> https://db.apache.org/derby/docs/10.8/devguide/cdevdvlpinmemdb.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-14476) Fix logging issue for branch-1

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868772#comment-15868772
 ] 

Hive QA commented on HIVE-14476:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12835870/HIVE-14476.1-branch-1.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 134 failed/errored test(s), 7899 tests 
executed
*Failed tests:*
{noformat}
TestAdminUser - did not produce a TEST-*.xml file (likely timed out) 
(batchId=340)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely 
timed out) (batchId=371)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely 
timed out) (batchId=350)
TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely 
timed out) (batchId=356)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out) 
(batchId=378)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed 
out) (batchId=394)
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely 
timed out) (batchId=370)
TestCompactor - did not produce a TEST-*.xml file (likely timed out) 
(batchId=360)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out) 
(batchId=359)
TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out) 
(batchId=379)
TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out) 
(batchId=325)
TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file 
(likely timed out) (batchId=358)
TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out) 
(batchId=328)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed 
out) (batchId=337)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely 
timed out) (batchId=382)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out) 
(batchId=332)
TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out) 
(batchId=365)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out) 
(batchId=397)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed 
out) (batchId=398)
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely 
timed out) (batchId=386)
TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely 
timed out) (batchId=374)
TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed 
out) (batchId=373)
TestHiveHistory - did not produce a TEST-*.xml file (likely timed out) 
(batchId=376)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out) 
(batchId=352)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file 
(likely timed out) (batchId=342)
TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out) 
(batchId=355)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out) 
(batchId=400)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed 
out) (batchId=401)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out) 
(batchId=383)
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out) 
(batchId=357)
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out) 
(batchId=429)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out) 
(batchId=388)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out) 
(batchId=399)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed 
out) (batchId=393)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out) 
(batchId=390)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out) 
(batchId=426)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed 
out) (batchId=425)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely 
timed out) (batchId=423)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely 
timed out) (batchId=428)
TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out) 
(batchId=389)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely 
timed out) (batchId=395)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed 
out) (batchId=396)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out) 
(batchId=363)
TestMTQueries - did not produce a TEST-*.xml file (likely timed out) 
(batchId=361)
TestMarkPartition - did not produce a TEST-*.xml file (likely timed out) 
(batchId=349)
TestMarkPartitionRemote - did not produce a TEST-*.xml file (likely timed out) 
(batchId=353)
TestMetaStoreAuthorization - did not produce a TEST-*.xml file (likely timed 
out) 

[jira] [Commented] (HIVE-15923) Hive default partition causes errors in get partitions

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868710#comment-15868710
 ] 

Aihua Xu commented on HIVE-15923:
-

[~sershe] Sure. I will take a look.

> Hive default partition causes errors in get partitions
> --
>
> Key: HIVE-15923
> URL: https://issues.apache.org/jira/browse/HIVE-15923
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15923.01.patch, HIVE-15923.patch
>
>
> This is the ORM error, direct SQL fails too before that, with a similar error.
> {noformat}
> 2017-02-14T17:45:11,158 ERROR [09fdd887-0164-4f55-97e9-4ba147d962be main] 
> metastore.ObjectStore:java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDefaultDesc cannot be cast to 
> java.lang.Long
> at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaLongObjectInspector.get(JavaLongObjectInspector.java:40)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getDouble(PrimitiveObjectInspectorUtils.java:801)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DoubleConverter.convert(P
> rimitiveObjectInspectorConverter.java:240) 
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqualOrGreaterThan.evaluate(GenericUDFOPEqualOrGreaterThan.java:145)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween.evaluate(GenericUDFBetween.java:57)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:88)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:63)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.evaluateExprOnPart(PartExprEvalUtils.java:126)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15673) Allow multiple queries with disjunction

[Vineet Garg (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868709#comment-15868709
 ] 

Vineet Garg commented on HIVE-15673:


TPC-DS query 10 & query 45 depends upon this

> Allow multiple queries with disjunction
> ---
>
> Key: HIVE-15673
> URL: https://issues.apache.org/jira/browse/HIVE-15673
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-15673.1.patch, HIVE-15673.2.patch, 
> HIVE-15673.3.patch, HIVE-15673.4.patch
>
>
> HIVE currently doesn't allow multiple subqueries with {{OR}} since calcite 
> has a bug in determining logic for OR expression. See [CALCITE-1546 
> |https://issues.apache.org/jira/browse/CALCITE-1546].
> Once calcite is released containing fix for the bug HIVE will need to lift 
> the restriction and add tests cases



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15910) Improvements in Hive Unit Test by using In-memory Derby DB

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15910:
-
Status: Patch Available  (was: Open)

> Improvements in Hive Unit Test by using In-memory Derby DB
> --
>
> Key: HIVE-15910
> URL: https://issues.apache.org/jira/browse/HIVE-15910
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Sankar Hariappan
>Assignee: Wei Zheng
> Attachments: HIVE-15910.01.patch, HIVE-15910.2.patch, 
> HIVE-15910.3.patch, HIVE-15910.4.patch
>
>
> Hive UT currently uses Derby DB with storage on disk which have some 
> practical problems.
> 1. The run-time of Hive unit tests are high as need to operate on the disk 
> quite often.
> 2. It can cause conflict if multiple test cases operates on the same table 
> name (such as table being created already exist).
> To solve these problems, we shall use an in-memory storage option of Derby DB 
> which can be even persisted if the test case demands that.
> https://db.apache.org/derby/docs/10.8/devguide/cdevdvlpinmemdb.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15910) Improvements in Hive Unit Test by using In-memory Derby DB

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15910:
-
Attachment: HIVE-15910.4.patch

> Improvements in Hive Unit Test by using In-memory Derby DB
> --
>
> Key: HIVE-15910
> URL: https://issues.apache.org/jira/browse/HIVE-15910
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Sankar Hariappan
>Assignee: Wei Zheng
> Attachments: HIVE-15910.01.patch, HIVE-15910.2.patch, 
> HIVE-15910.3.patch, HIVE-15910.4.patch
>
>
> Hive UT currently uses Derby DB with storage on disk which have some 
> practical problems.
> 1. The run-time of Hive unit tests are high as need to operate on the disk 
> quite often.
> 2. It can cause conflict if multiple test cases operates on the same table 
> name (such as table being created already exist).
> To solve these problems, we shall use an in-memory storage option of Derby DB 
> which can be even persisted if the test case demands that.
> https://db.apache.org/derby/docs/10.8/devguide/cdevdvlpinmemdb.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15910) Improvements in Hive Unit Test by using In-memory Derby DB

[Wei Zheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei Zheng updated HIVE-15910:
-
Status: Open  (was: Patch Available)

> Improvements in Hive Unit Test by using In-memory Derby DB
> --
>
> Key: HIVE-15910
> URL: https://issues.apache.org/jira/browse/HIVE-15910
> Project: Hive
>  Issue Type: Test
>  Components: Tests
>Reporter: Sankar Hariappan
>Assignee: Wei Zheng
> Attachments: HIVE-15910.01.patch, HIVE-15910.2.patch, 
> HIVE-15910.3.patch
>
>
> Hive UT currently uses Derby DB with storage on disk which have some 
> practical problems.
> 1. The run-time of Hive unit tests are high as need to operate on the disk 
> quite often.
> 2. It can cause conflict if multiple test cases operates on the same table 
> name (such as table being created already exist).
> To solve these problems, we shall use an in-memory storage option of Derby DB 
> which can be even persisted if the test case demands that.
> https://db.apache.org/derby/docs/10.8/devguide/cdevdvlpinmemdb.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15923) Hive default partition causes errors in get partitions

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868656#comment-15868656
 ] 

Sergey Shelukhin commented on HIVE-15923:
-

[~aihuaxu] can you take a look? Thanks

> Hive default partition causes errors in get partitions
> --
>
> Key: HIVE-15923
> URL: https://issues.apache.org/jira/browse/HIVE-15923
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15923.01.patch, HIVE-15923.patch
>
>
> This is the ORM error, direct SQL fails too before that, with a similar error.
> {noformat}
> 2017-02-14T17:45:11,158 ERROR [09fdd887-0164-4f55-97e9-4ba147d962be main] 
> metastore.ObjectStore:java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDefaultDesc cannot be cast to 
> java.lang.Long
> at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaLongObjectInspector.get(JavaLongObjectInspector.java:40)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getDouble(PrimitiveObjectInspectorUtils.java:801)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DoubleConverter.convert(P
> rimitiveObjectInspectorConverter.java:240) 
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqualOrGreaterThan.evaluate(GenericUDFOPEqualOrGreaterThan.java:145)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween.evaluate(GenericUDFBetween.java:57)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:88)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:63)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.evaluateExprOnPart(PartExprEvalUtils.java:126)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15923) Hive default partition causes errors in get partitions

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15923:

Attachment: HIVE-15923.01.patch

Fixing the original test via a custom UDF. I think it might be possible to make 
it more explicit by creating a different evaluator for the column expr, that 
would return null in case of type incompatibility, used only for drop; but it 
amounts to the same thing.

> Hive default partition causes errors in get partitions
> --
>
> Key: HIVE-15923
> URL: https://issues.apache.org/jira/browse/HIVE-15923
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15923.01.patch, HIVE-15923.patch
>
>
> This is the ORM error, direct SQL fails too before that, with a similar error.
> {noformat}
> 2017-02-14T17:45:11,158 ERROR [09fdd887-0164-4f55-97e9-4ba147d962be main] 
> metastore.ObjectStore:java.lang.ClassCastException: 
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDefaultDesc cannot be cast to 
> java.lang.Long
> at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaLongObjectInspector.get(JavaLongObjectInspector.java:40)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getDouble(PrimitiveObjectInspectorUtils.java:801)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]at 
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DoubleConverter.convert(P
> rimitiveObjectInspectorConverter.java:240) 
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqualOrGreaterThan.evaluate(GenericUDFOPEqualOrGreaterThan.java:145)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween.evaluate(GenericUDFBetween.java:57)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:88)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:63)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.evaluateExprOnPart(PartExprEvalUtils.java:126)
>  ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background

[Mohit Sabharwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868655#comment-15868655
 ] 

Mohit Sabharwal commented on HIVE-15710:


[~Ferd], proposal to only do background logic for beeline and hs2 looks good to 
me.

> HS2 Stopped when running in background
> --
>
> Key: HIVE-15710
> URL: https://issues.apache.org/jira/browse/HIVE-15710
> Project: Hive
>  Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15710.1.patch, HIVE-15710.2.patch
>
>
> To reproduce, start HS2 in background like {{hive --service hiveserver2 &}}, 
> and run some query from beeline using Tez or Spark as engine.
> I think it's similar to HIVE-6758: HS2 uses jline for the in-place progress 
> update, and receives SIGTTOU when trying to call stty.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-15915) Emit progress percentage in getting operation status

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868631#comment-15868631
 ] 

Hive QA commented on HIVE-15915:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12852870/HIVE-15915.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10224 tests 
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) 
(batchId=235)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] 
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join31] (batchId=81)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multiMapJoin2]
 (batchId=152)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] 
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] 
(batchId=223)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
 (batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join31] 
(batchId=133)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3574/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3574/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3574/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12852870 - PreCommit-HIVE-Build

> Emit progress percentage in getting operation status
> 
>
> Key: HIVE-15915
> URL: https://issues.apache.org/jira/browse/HIVE-15915
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Minor
> Attachments: HIVE-15915.1.patch
>
>
> When running a query asynchronously, client may want to check the progress 
> periodically. HIVE-15473 is to support progressing bar on beeline for Tez. 
> For this issue, we just want the progress percentage. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (HIVE-1555) JDBC Storage Handler

[Gunther Hagleitner (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868626#comment-15868626
 ] 

Gunther Hagleitner commented on HIVE-1555:
--

Test failures aren't related. [~ashutoshc] do you have some feedback?

> JDBC Storage Handler
> 
>
> Key: HIVE-1555
> URL: https://issues.apache.org/jira/browse/HIVE-1555
> Project: Hive
>  Issue Type: New Feature
>  Components: JDBC
>Reporter: Bob Robertson
>Assignee: Gunther Hagleitner
> Attachments: HIVE-1555.3.patch, HIVE-1555.4.patch, HIVE-1555.5.patch, 
> JDBCStorageHandler Design Doc.pdf
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> With the Cassandra and HBase Storage Handlers I thought it would make sense 
> to include a generic JDBC RDBMS Storage Handler so that you could import a 
> standard DB table into Hive. Many people must want to perform HiveQL joins, 
> etc against tables in other systems etc.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15929) Fix HiveDecimalWritable to be compatible with Hive 2.1

[Owen O'Malley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Owen O'Malley updated HIVE-15929:
-
Attachment: HIVE-15929.patch

Matches the pull request.

> Fix HiveDecimalWritable to be compatible with Hive 2.1
> --
>
> Key: HIVE-15929
> URL: https://issues.apache.org/jira/browse/HIVE-15929
> Project: Hive
>  Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-15929.patch
>
>
> HIVE-15335 broke compatibility with Hive 2.1 by making 
> HiveDecimalWritable.getInternalStorate() throw an exception when called on an 
> unset value. It is easy to instead return an empty array, which will allow 
> the old code to allocate a new array.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)