[jira] [Updated] (HIVE-16282) Semijoin: Disable slow-start for the bloom filter aggregate task
[ https://issues.apache.org/jira/browse/HIVE-16282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-16282: -- Attachment: HIVE-16282.2.patch > Semijoin: Disable slow-start for the bloom filter aggregate task > > > Key: HIVE-16282 > URL: https://issues.apache.org/jira/browse/HIVE-16282 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Deepak Jaiswal > Attachments: extended plan.rtf, HIVE-16282.1.patch, HIVE-16282.2.patch > > > The slow-start of the bloom filter vertex is a scheduling problem which > causes more pre-emption than is useful. > When the bloom filters are arranged as follows > Map 1(10 tasks)->Reducer 2(1 task)->Map 3(100 tasks) > Map 3 and Map 1 are immediately active since Reducer 2 -> Map 3 is a > broadcast edge. > Once 3 tasks in Map 1 finish, the engine kills one active task from Map 3 to > make room for Reducer 2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16282) Semijoin: Disable slow-start for the bloom filter aggregate task
[
https://issues.apache.org/jira/browse/HIVE-16282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939913#comment-15939913
]
Hive QA commented on HIVE-16282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860279/extended%20plan.rtf
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4334/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4334/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4334/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-24 06:34:12.271
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4334/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-24 06:34:12.275
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b4d1f96 HIVE-15841: Upgrade Hive to ORC 1.3.3 (Owen O'Malley
reviewed by Prasanth Jayachandran)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b4d1f96 HIVE-15841: Upgrade Hive to ORC 1.3.3 (Owen O'Malley
reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-24 06:34:13.143
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860279 - PreCommit-HIVE-Build
> Semijoin: Disable slow-start for the bloom filter aggregate task
>
>
> Key: HIVE-16282
> URL: https://issues.apache.org/jira/browse/HIVE-16282
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Deepak Jaiswal
> Attachments: extended plan.rtf, HIVE-16282.1.patch
>
>
> The slow-start of the bloom filter vertex is a scheduling problem which
> causes more pre-emption than is useful.
> When the bloom filters are arranged as follows
> Map 1(10 tasks)->Reducer 2(1 task)->Map 3(100 tasks)
> Map 3 and Map 1 are immediately active since Reducer 2 -> Map 3 is a
> broadcast edge.
> Once 3 tasks in Map 1 finish, the engine kills one active task from Map 3 to
> make room for Reducer 2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15644) Collect LLAP's JVM metrics via Hive's JvmPauseMonitor
[
https://issues.apache.org/jira/browse/HIVE-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939910#comment-15939910
]
Hive QA commented on HIVE-15644:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860242/HIVE-15644.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10511 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_count_distinct]
(batchId=107)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4332/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4332/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4332/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860242 - PreCommit-HIVE-Build
> Collect LLAP's JVM metrics via Hive's JvmPauseMonitor
> -
>
> Key: HIVE-15644
> URL: https://issues.apache.org/jira/browse/HIVE-15644
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Labels: llap
> Attachments: HIVE-15644.1.patch
>
>
> Similar to what Hadoop's JvmMetrics is doing
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16282) Semijoin: Disable slow-start for the bloom filter aggregate task
[
https://issues.apache.org/jira/browse/HIVE-16282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939912#comment-15939912
]
Hive QA commented on HIVE-16282:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860279/extended%20plan.rtf
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4333/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4333/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4333/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-24 06:33:37.711
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4333/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-24 06:33:37.714
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b4d1f96 HIVE-15841: Upgrade Hive to ORC 1.3.3 (Owen O'Malley
reviewed by Prasanth Jayachandran)
+ git clean -f -d
Removing common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java
Removing common/src/java/org/apache/hadoop/hive/common/JvmMetricsInfo.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b4d1f96 HIVE-15841: Upgrade Hive to ORC 1.3.3 (Owen O'Malley
reviewed by Prasanth Jayachandran)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-24 06:33:39.039
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
patch: Only garbage was found in the patch input.
fatal: unrecognized input
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860279 - PreCommit-HIVE-Build
> Semijoin: Disable slow-start for the bloom filter aggregate task
>
>
> Key: HIVE-16282
> URL: https://issues.apache.org/jira/browse/HIVE-16282
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Deepak Jaiswal
> Attachments: extended plan.rtf, HIVE-16282.1.patch
>
>
> The slow-start of the bloom filter vertex is a scheduling problem which
> causes more pre-emption than is useful.
> When the bloom filters are arranged as follows
> Map 1(10 tasks)->Reducer 2(1 task)->Map 3(100 tasks)
> Map 3 and Map 1 are immediately active since Reducer 2 -> Map 3 is a
> broadcast edge.
> Once 3 tasks in Map 1 finish, the engine kills one active task from Map 3 to
> make room for Reducer 2.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15880) Allow insert overwrite query to use auto.purge table property
[
https://issues.apache.org/jira/browse/HIVE-15880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-15880:
---
Attachment: HIVE-15880.03.patch
The tests were working for me locally. Changed the implementation of the tests
to make it more robust. Also, added more logs in the test case to help
debugging in case it fails again. Added couple of more tests
> Allow insert overwrite query to use auto.purge table property
> -
>
> Key: HIVE-15880
> URL: https://issues.apache.org/jira/browse/HIVE-15880
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-15880.01.patch, HIVE-15880.02.patch,
> HIVE-15880.03.patch
>
>
> It seems inconsistent that auto.purge property is not considered when we do a
> INSERT OVERWRITE while it is when we do a DROP TABLE
> Drop table doesn't move table data to Trash when auto.purge is set to true
> {noformat}
> > create table temp(col1 string, col2 string);
> No rows affected (0.064 seconds)
> > alter table temp set tblproperties('auto.purge'='true');
> No rows affected (0.083 seconds)
> > insert into temp values ('test', 'test'), ('test2', 'test2');
> No rows affected (25.473 seconds)
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 22 2017-02-09 13:03
> /user/hive/warehouse/temp/00_0
> #
> > drop table temp;
> No rows affected (0.242 seconds)
> # hdfs dfs -ls /user/hive/warehouse/temp
> ls: `/user/hive/warehouse/temp': No such file or directory
> #
> # sudo -u hive hdfs dfs -ls /user/hive/.Trash/Current/user/hive/warehouse
> #
> {noformat}
> INSERT OVERWRITE query moves the table data to Trash even when auto.purge is
> set to true
> {noformat}
> > create table temp(col1 string, col2 string);
> > alter table temp set tblproperties('auto.purge'='true');
> > insert into temp values ('test', 'test'), ('test2', 'test2');
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 22 2017-02-09 13:07
> /user/hive/warehouse/temp/00_0
> #
> > insert overwrite table temp select * from dummy;
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 26 2017-02-09 13:08
> /user/hive/warehouse/temp/00_0
> # sudo -u hive hdfs dfs -ls /user/hive/.Trash/Current/user/hive/warehouse
> Found 1 items
> drwx-- - hive hive 0 2017-02-09 13:08
> /user/hive/.Trash/Current/user/hive/warehouse/temp
> #
> {noformat}
> While move operations are not very costly on HDFS it could be significant
> overhead on slow FileSystems like S3. This could improve the performance of
> {{INSERT OVERWRITE TABLE}} queries especially when there are large number of
> partitions on tables located on S3 should the user wish to set auto.purge
> property to true
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16285) Servlet for dynamically configuring log levels
[
https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939842#comment-15939842
]
Hive QA commented on HIVE-16285:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860251/HIVE-16285.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10511 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4331/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4331/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4331/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860251 - PreCommit-HIVE-Build
> Servlet for dynamically configuring log levels
> --
>
> Key: HIVE-16285
> URL: https://issues.apache.org/jira/browse/HIVE-16285
> Project: Hive
> Issue Type: Improvement
> Components: Logging
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16285.1.patch, HIVE-16285.2.patch
>
>
> Many long running services like HS2, LLAP etc. will benefit from having an
> endpoint to dynamically change log levels for various loggers. This will help
> greatly with debuggability without requiring a restart of the service.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15841) Upgrade Hive to ORC 1.3.3
[ https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-15841: - Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master. > Upgrade Hive to ORC 1.3.3 > - > > Key: HIVE-15841 > URL: https://issues.apache.org/jira/browse/HIVE-15841 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.2.0 > > Attachments: HIVE-15841.2.patch, HIVE-15841.patch, HIVE-15841.patch, > HIVE-15841.patch > > > Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.3 once it > releases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15841) Upgrade Hive to ORC 1.3.3
[ https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939805#comment-15939805 ] Prasanth Jayachandran commented on HIVE-15841: -- Other test failures are already happening in master. > Upgrade Hive to ORC 1.3.3 > - > > Key: HIVE-15841 > URL: https://issues.apache.org/jira/browse/HIVE-15841 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.2.0 > > Attachments: HIVE-15841.2.patch, HIVE-15841.patch, HIVE-15841.patch, > HIVE-15841.patch > > > Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.3 once it > releases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16254) metadata for values temporary tables for INSERT's are getting replicated
[ https://issues.apache.org/jira/browse/HIVE-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939801#comment-15939801 ] anishek edited comment on HIVE-16254 at 3/24/17 5:04 AM: - [~thejas]/[~vgumashta] please review this. the failure test has been failing for sometime and is not related to the current set of changes. was (Author: anishek): [~thejas]/[~vgumashta] please review this. > metadata for values temporary tables for INSERT's are getting replicated > > > Key: HIVE-16254 > URL: https://issues.apache.org/jira/browse/HIVE-16254 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.2.0 >Reporter: anishek >Assignee: anishek > Attachments: HIVE-16254.3.patch > > > create table a (age int); > insert into table a values (34),(4); > repl dump default; > there is a temporary table created as values__tmp__table__[nmber], which is > also present in the dumped information with only metadata, this should not be > processed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16254) metadata for values temporary tables for INSERT's are getting replicated
[ https://issues.apache.org/jira/browse/HIVE-16254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939801#comment-15939801 ] anishek commented on HIVE-16254: [~thejas]/[~vgumashta] please review this. > metadata for values temporary tables for INSERT's are getting replicated > > > Key: HIVE-16254 > URL: https://issues.apache.org/jira/browse/HIVE-16254 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.2.0 >Reporter: anishek >Assignee: anishek > Attachments: HIVE-16254.3.patch > > > create table a (age int); > insert into table a values (34),(4); > repl dump default; > there is a temporary table created as values__tmp__table__[nmber], which is > also present in the dumped information with only metadata, this should not be > processed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15841) Upgrade Hive to ORC 1.3.3
[
https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939780#comment-15939780
]
Hive QA commented on HIVE-15841:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860247/HIVE-15841.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10511 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4330/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4330/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4330/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860247 - PreCommit-HIVE-Build
> Upgrade Hive to ORC 1.3.3
> -
>
> Key: HIVE-15841
> URL: https://issues.apache.org/jira/browse/HIVE-15841
> Project: Hive
> Issue Type: Bug
>Reporter: Owen O'Malley
>Assignee: Owen O'Malley
> Attachments: HIVE-15841.2.patch, HIVE-15841.patch, HIVE-15841.patch,
> HIVE-15841.patch
>
>
> Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.3 once it
> releases.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15880) Allow insert overwrite query to use auto.purge table property
[
https://issues.apache.org/jira/browse/HIVE-15880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939745#comment-15939745
]
Hive QA commented on HIVE-15880:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860246/HIVE-15880.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10519 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testPartitionedExternalNoAutoPurge
(batchId=211)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testPartitionedNoAutoPurge
(batchId=211)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4329/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4329/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4329/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860246 - PreCommit-HIVE-Build
> Allow insert overwrite query to use auto.purge table property
> -
>
> Key: HIVE-15880
> URL: https://issues.apache.org/jira/browse/HIVE-15880
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-15880.01.patch, HIVE-15880.02.patch
>
>
> It seems inconsistent that auto.purge property is not considered when we do a
> INSERT OVERWRITE while it is when we do a DROP TABLE
> Drop table doesn't move table data to Trash when auto.purge is set to true
> {noformat}
> > create table temp(col1 string, col2 string);
> No rows affected (0.064 seconds)
> > alter table temp set tblproperties('auto.purge'='true');
> No rows affected (0.083 seconds)
> > insert into temp values ('test', 'test'), ('test2', 'test2');
> No rows affected (25.473 seconds)
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 22 2017-02-09 13:03
> /user/hive/warehouse/temp/00_0
> #
> > drop table temp;
> No rows affected (0.242 seconds)
> # hdfs dfs -ls /user/hive/warehouse/temp
> ls: `/user/hive/warehouse/temp': No such file or directory
> #
> # sudo -u hive hdfs dfs -ls /user/hive/.Trash/Current/user/hive/warehouse
> #
> {noformat}
> INSERT OVERWRITE query moves the table data to Trash even when auto.purge is
> set to true
> {noformat}
> > create table temp(col1 string, col2 string);
> > alter table temp set tblproperties('auto.purge'='true');
> > insert into temp values ('test', 'test'), ('test2', 'test2');
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 22 2017-02-09 13:07
> /user/hive/warehouse/temp/00_0
> #
> > insert overwrite table temp select * from dummy;
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 26 2017-02-09 13:08
> /user/hive/warehouse/temp/00_0
> # sudo -u hive hdfs dfs -ls /user/hive/.Trash/Current/user/hive/warehouse
> Found 1 items
> drwx-- - hive hive 0 2017-02-09 13:08
> /user/hive/.Trash/Current/user/hive/warehouse/temp
> #
> {noformat}
> While move operations are not very costly on HDFS it could be significant
> overhead on slow FileSystems like S3. This could improve the performance of
> {{INSERT OVERWRITE TABLE}} queries especially when there are large number of
> partitions on tables located on S3 should the user wish to set auto.purge
> property to true
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16282) Semijoin: Disable slow-start for the bloom filter aggregate task
[ https://issues.apache.org/jira/browse/HIVE-16282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-16282: -- Attachment: extended plan.rtf Attaching an extended plan. Please let me know if this is what is expected. > Semijoin: Disable slow-start for the bloom filter aggregate task > > > Key: HIVE-16282 > URL: https://issues.apache.org/jira/browse/HIVE-16282 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Deepak Jaiswal > Attachments: extended plan.rtf, HIVE-16282.1.patch > > > The slow-start of the bloom filter vertex is a scheduling problem which > causes more pre-emption than is useful. > When the bloom filters are arranged as follows > Map 1(10 tasks)->Reducer 2(1 task)->Map 3(100 tasks) > Map 3 and Map 1 are immediately active since Reducer 2 -> Map 3 is a > broadcast edge. > Once 3 tasks in Map 1 finish, the engine kills one active task from Map 3 to > make room for Reducer 2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16233) llap: Query failed with AllocatorOutOfMemoryException
[
https://issues.apache.org/jira/browse/HIVE-16233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939634#comment-15939634
]
Sergey Shelukhin commented on HIVE-16233:
-
Might have a patch tomorrow or next week...
> llap: Query failed with AllocatorOutOfMemoryException
> -
>
> Key: HIVE-16233
> URL: https://issues.apache.org/jira/browse/HIVE-16233
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
>
> {code}
> TaskAttempt 5 failed, info=[Error: Error while running task ( failure ) :
> attempt_1488231257387_2288_25_05_56_5:java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> java.io.IOException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 262144; at 0 out of 1
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.IOException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 262144; at 0 out of 1
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:74)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: java.io.IOException: java.io.IOException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 262144; at 0 out of 1
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
> at
> org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
> at
> org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
> at
> org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:151)
> at
> org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:62)
> ... 17 more
> Caused by: java.io.IOException:
> org.apache.hadoop.hive.common.io.Allocator$AllocatorOutOfMemoryException:
> Failed to allocate 262144; at 0 out of 1
> at
> org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReaderImpl.readEncodedColumns(EncodedReaderImpl.java:425)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:413)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:235)
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:232)
[jira] [Commented] (HIVE-16239) remove useless hiveserver
[
https://issues.apache.org/jira/browse/HIVE-16239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939631#comment-15939631
]
Ferdinand Xu commented on HIVE-16239:
-
[~ferhui] We need to wait for 24 hours to commit a patch after +1 from a
committer. This is in case someone else has further comments about that patch.
> remove useless hiveserver
> -
>
> Key: HIVE-16239
> URL: https://issues.apache.org/jira/browse/HIVE-16239
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 2.0.1, 2.1.1
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-16239.1-branch-2.0.patch,
> HIVE-16239.1-branch-2.1.patch
>
>
> {quote}
> [hadoop@header hive]$ hive --service hiveserver
> Starting Hive Thrift Server
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/apps/apache-hive-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/apps/spark-1.6.2-bin-hadoop2.7/lib/spark-assembly-1.6.2-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/apps/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.service.HiveServer
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {quote}
> hiveserver does not exist, we should remove hiveserver from cli on branch-2.0
> After removing it, we get useful message
> {quote}
> Service hiveserver not found
> Available Services: beeline cli hbaseimport hbaseschematool help
> hiveburninclient hiveserver2 hplsql hwi jar lineage llap metastore metatool
> orcfiledump rcfilecat schemaTool version
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16239) remove useless hiveserver
[
https://issues.apache.org/jira/browse/HIVE-16239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939623#comment-15939623
]
Fei Hui commented on HIVE-16239:
[~Ferd] Could you please commit it ? Thanks.
> remove useless hiveserver
> -
>
> Key: HIVE-16239
> URL: https://issues.apache.org/jira/browse/HIVE-16239
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 2.0.1, 2.1.1
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-16239.1-branch-2.0.patch,
> HIVE-16239.1-branch-2.1.patch
>
>
> {quote}
> [hadoop@header hive]$ hive --service hiveserver
> Starting Hive Thrift Server
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/apps/apache-hive-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/apps/spark-1.6.2-bin-hadoop2.7/lib/spark-assembly-1.6.2-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/apps/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.service.HiveServer
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {quote}
> hiveserver does not exist, we should remove hiveserver from cli on branch-2.0
> After removing it, we get useful message
> {quote}
> Service hiveserver not found
> Available Services: beeline cli hbaseimport hbaseschematool help
> hiveburninclient hiveserver2 hplsql hwi jar lineage llap metastore metatool
> orcfiledump rcfilecat schemaTool version
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16286) Log canceled query id
[ https://issues.apache.org/jira/browse/HIVE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-16286: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Test failures don't seem related to the patch here. Committed to master. Thanks to Jimmy for the contribution. > Log canceled query id > - > > Key: HIVE-16286 > URL: https://issues.apache.org/jira/browse/HIVE-16286 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Fix For: 2.2.0 > > Attachments: HIVE-16286.1.patch > > > Currently, just a generic message is logged when a query is canceled. It is > better to log the query id as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16289) add hints for semijoin reduction
[
https://issues.apache.org/jira/browse/HIVE-16289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939620#comment-15939620
]
Hive QA commented on HIVE-16289:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860236/HIVE-16289.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4328/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4328/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4328/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-03-24 02:48:48.903
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4328/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-03-24 02:48:48.906
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
736d2e8..8fa9d58 master -> origin/master
+ git reset --hard HEAD
HEAD is now at 736d2e8 HIVE-16154: Determine when dynamic runtime filtering
should be disabled (Jason Dere, reviewed by Gunther Hagleitner)
+ git clean -f -d
Removing data/files/events.txt
Removing itests/hive-blobstore/src/test/queries/clientpositive/orc_buckets.q
Removing
itests/hive-blobstore/src/test/queries/clientpositive/orc_format_nonpart.q
Removing itests/hive-blobstore/src/test/queries/clientpositive/orc_format_part.q
Removing
itests/hive-blobstore/src/test/queries/clientpositive/orc_nonstd_partitions_loc.q
Removing itests/hive-blobstore/src/test/queries/clientpositive/rcfile_buckets.q
Removing
itests/hive-blobstore/src/test/queries/clientpositive/rcfile_format_nonpart.q
Removing
itests/hive-blobstore/src/test/queries/clientpositive/rcfile_format_part.q
Removing
itests/hive-blobstore/src/test/queries/clientpositive/rcfile_nonstd_partitions_loc.q
Removing itests/hive-blobstore/src/test/results/clientpositive/orc_buckets.q.out
Removing
itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out
Removing
itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out
Removing
itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out
Removing
itests/hive-blobstore/src/test/results/clientpositive/rcfile_buckets.q.out
Removing
itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out
Removing
itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out
Removing
itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 8fa9d58 HIVE-16286: Log canceled query id (Jimmy via Xuefu)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-03-24 02:48:50.799
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java:504
error:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java:
patch does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860236 - PreCommit-HIVE-Build
> add hints for semijoin reduction
>
>
> Key: HIVE-16289
> URL: https://issues.apache.org/jira/browse/HIVE-16289
> Project: Hive
> Issue Type: Bug
>
[jira] [Commented] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[
https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939618#comment-15939618
]
liyunzhang_intel commented on HIVE-13517:
-
[~stakiar]: LGTM, but what i am confused
{quote}
In a future patch, we could add a better default log4j properties file, which
has the thread-id.
{quote}
Based on HIVE-13517.2.patch, if users want a specified pattern(like contains
thread-id) in log4j.properties, they can write the pattern in
{{hive.spark.log4j.file}}, why need add a better default log4j.properties later?
> Hive logs in Spark Executor and Driver should show thread-id.
> -
>
> Key: HIVE-13517
> URL: https://issues.apache.org/jira/browse/HIVE-13517
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Szehon Ho
>Assignee: Sahil Takiar
> Attachments: executor-driver-log.PNG, HIVE-13517.1.patch,
> HIVE-13517.2.patch
>
>
> In Spark, there might be more than one task running in one executor.
> Similarly, there may be more than one thread running in Driver.
> This makes debugging through the logs a nightmare. It would be great if there
> could be thread-ids in the logs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16288) Add blobstore tests for ORC and RCFILE file formats
[
https://issues.apache.org/jira/browse/HIVE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939617#comment-15939617
]
Hive QA commented on HIVE-16288:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860234/HIVE-16288.patch
{color:green}SUCCESS:{color} +1 due to 8 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10519 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4327/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4327/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4327/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860234 - PreCommit-HIVE-Build
> Add blobstore tests for ORC and RCFILE file formats
> ---
>
> Key: HIVE-16288
> URL: https://issues.apache.org/jira/browse/HIVE-16288
> Project: Hive
> Issue Type: Test
> Components: Tests
>Affects Versions: 2.1.1
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
> Attachments: HIVE-16288.patch
>
>
> This patch adds four tests each for ORC and RCFILE when running against
> blobstore filesystems:
> * Test for bucketed tables
> * Test for nonpartitioned tables
> * Test for partitioned tables
> * Test for partitioned tables with nonstandard partition locations
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16282) Semijoin: Disable slow-start for the bloom filter aggregate task
[ https://issues.apache.org/jira/browse/HIVE-16282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939615#comment-15939615 ] Deepak Jaiswal commented on HIVE-16282: --- RB link, https://reviews.apache.org/r/57903/ > Semijoin: Disable slow-start for the bloom filter aggregate task > > > Key: HIVE-16282 > URL: https://issues.apache.org/jira/browse/HIVE-16282 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Deepak Jaiswal > Attachments: HIVE-16282.1.patch > > > The slow-start of the bloom filter vertex is a scheduling problem which > causes more pre-emption than is useful. > When the bloom filters are arranged as follows > Map 1(10 tasks)->Reducer 2(1 task)->Map 3(100 tasks) > Map 3 and Map 1 are immediately active since Reducer 2 -> Map 3 is a > broadcast edge. > Once 3 tasks in Map 1 finish, the engine kills one active task from Map 3 to > make room for Reducer 2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16282) Semijoin: Disable slow-start for the bloom filter aggregate task
[ https://issues.apache.org/jira/browse/HIVE-16282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939590#comment-15939590 ] Gopal V commented on HIVE-16282: [~djaiswal]: minor nit - this applies to simple_edge as well (if this gets backported to a branch without HIVE-14707). Otherwise LGTM - +1 tests pending. > Semijoin: Disable slow-start for the bloom filter aggregate task > > > Key: HIVE-16282 > URL: https://issues.apache.org/jira/browse/HIVE-16282 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Deepak Jaiswal > Attachments: HIVE-16282.1.patch > > > The slow-start of the bloom filter vertex is a scheduling problem which > causes more pre-emption than is useful. > When the bloom filters are arranged as follows > Map 1(10 tasks)->Reducer 2(1 task)->Map 3(100 tasks) > Map 3 and Map 1 are immediately active since Reducer 2 -> Map 3 is a > broadcast edge. > Once 3 tasks in Map 1 finish, the engine kills one active task from Map 3 to > make room for Reducer 2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16282) Semijoin: Disable slow-start for the bloom filter aggregate task
[ https://issues.apache.org/jira/browse/HIVE-16282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-16282: -- Attachment: HIVE-16282.1.patch > Semijoin: Disable slow-start for the bloom filter aggregate task > > > Key: HIVE-16282 > URL: https://issues.apache.org/jira/browse/HIVE-16282 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Deepak Jaiswal > Attachments: HIVE-16282.1.patch > > > The slow-start of the bloom filter vertex is a scheduling problem which > causes more pre-emption than is useful. > When the bloom filters are arranged as follows > Map 1(10 tasks)->Reducer 2(1 task)->Map 3(100 tasks) > Map 3 and Map 1 are immediately active since Reducer 2 -> Map 3 is a > broadcast edge. > Once 3 tasks in Map 1 finish, the engine kills one active task from Map 3 to > make room for Reducer 2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16282) Semijoin: Disable slow-start for the bloom filter aggregate task
[ https://issues.apache.org/jira/browse/HIVE-16282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-16282: -- Status: Patch Available (was: In Progress) > Semijoin: Disable slow-start for the bloom filter aggregate task > > > Key: HIVE-16282 > URL: https://issues.apache.org/jira/browse/HIVE-16282 > Project: Hive > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Gopal V >Assignee: Deepak Jaiswal > > The slow-start of the bloom filter vertex is a scheduling problem which > causes more pre-emption than is useful. > When the bloom filters are arranged as follows > Map 1(10 tasks)->Reducer 2(1 task)->Map 3(100 tasks) > Map 3 and Map 1 are immediately active since Reducer 2 -> Map 3 is a > broadcast edge. > Once 3 tasks in Map 1 finish, the engine kills one active task from Map 3 to > make room for Reducer 2. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16281) Upgrade master branch to JDK8
[
https://issues.apache.org/jira/browse/HIVE-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939566#comment-15939566
]
Vineet Garg commented on HIVE-16281:
TestCliDriver.testCliDriver[comments] is not related to HIVE-16229 changes. I
just ran test in HIVE-16229's branch to make sure and it is not failing in that
branch. HIVE-16229's change is only for subqueries and {{comments.q}} doesn't
have any query with subquery.
> Upgrade master branch to JDK8
> -
>
> Key: HIVE-16281
> URL: https://issues.apache.org/jira/browse/HIVE-16281
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-16281.1.patch, HIVE-16281.2.patch
>
>
> This is to track the JDK 8 upgrade work for the master branch.
> Here are threads for the discussion:
> https://lists.apache.org/thread.html/83d8235bc9547cc94a0d689580f20db4b946876b6d0369e31ea12b51@1460158490@%3Cdev.hive.apache.org%3E
> https://lists.apache.org/thread.html/dcd57844ceac7faf8975a00d5b8b1825ab5544d94734734aedc3840e@%3Cdev.hive.apache.org%3E
> JDK7 is end of public update and some newer version of dependent libraries
> like jetty require newer JDK. Seems it's reasonable to upgrade to JDK8 in 2.x.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16285) Servlet for dynamically configuring log levels
[
https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939557#comment-15939557
]
Hive QA commented on HIVE-16285:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860251/HIVE-16285.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10511 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=95)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4326/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4326/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4326/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860251 - PreCommit-HIVE-Build
> Servlet for dynamically configuring log levels
> --
>
> Key: HIVE-16285
> URL: https://issues.apache.org/jira/browse/HIVE-16285
> Project: Hive
> Issue Type: Improvement
> Components: Logging
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16285.1.patch, HIVE-16285.2.patch
>
>
> Many long running services like HS2, LLAP etc. will benefit from having an
> endpoint to dynamically change log levels for various loggers. This will help
> greatly with debuggability without requiring a restart of the service.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14602) NPE when running delete_orig_table.q,update_orig_table.q under MiniLlap
[
https://issues.apache.org/jira/browse/HIVE-14602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939522#comment-15939522
]
Prasanth Jayachandran commented on HIVE-14602:
--
These tests are currently running only in MiniTez. Haven't checked in awhile if
this happens now in MiniLlap or not.
> NPE when running delete_orig_table.q,update_orig_table.q under MiniLlap
> ---
>
> Key: HIVE-14602
> URL: https://issues.apache.org/jira/browse/HIVE-14602
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>
> When delete_orig_table.q,update_orig_table.q is run under
> TestMiniLlapCliDriver, NPE is thrown
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1471902873303_0001_31_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"reducesinkkey0":null},"value":null}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0) {"key":{"reducesinkkey0":null},"value":null}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:289)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:279)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":null},"value":null}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:357)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:279)
> ... 17 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:764)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)
> ... 18 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16281) Upgrade master branch to JDK8
[ https://issues.apache.org/jira/browse/HIVE-16281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939489#comment-15939489 ] Thejas M Nair commented on HIVE-16281: -- +1 TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] is tracked by HIVE-15776 [~vgarg] Acccording to above comment TestCliDriver.testCliDriver[comments] failure seems related to HIVE-16229 . Can you please take a look ? > Upgrade master branch to JDK8 > - > > Key: HIVE-16281 > URL: https://issues.apache.org/jira/browse/HIVE-16281 > Project: Hive > Issue Type: New Feature > Components: Hive >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-16281.1.patch, HIVE-16281.2.patch > > > This is to track the JDK 8 upgrade work for the master branch. > Here are threads for the discussion: > https://lists.apache.org/thread.html/83d8235bc9547cc94a0d689580f20db4b946876b6d0369e31ea12b51@1460158490@%3Cdev.hive.apache.org%3E > https://lists.apache.org/thread.html/dcd57844ceac7faf8975a00d5b8b1825ab5544d94734734aedc3840e@%3Cdev.hive.apache.org%3E > JDK7 is end of public update and some newer version of dependent libraries > like jetty require newer JDK. Seems it's reasonable to upgrade to JDK8 in 2.x. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14602) NPE when running delete_orig_table.q,update_orig_table.q under MiniLlap
[
https://issues.apache.org/jira/browse/HIVE-14602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939482#comment-15939482
]
Andy commented on HIVE-14602:
-
Hi Prasanth Jayachandran,
How did workaround this problem?
Or are you still suffering from it?
> NPE when running delete_orig_table.q,update_orig_table.q under MiniLlap
> ---
>
> Key: HIVE-14602
> URL: https://issues.apache.org/jira/browse/HIVE-14602
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>
> When delete_orig_table.q,update_orig_table.q is run under
> TestMiniLlapCliDriver, NPE is thrown
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1471902873303_0001_31_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"reducesinkkey0":null},"value":null}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0) {"key":{"reducesinkkey0":null},"value":null}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:289)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:279)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":null},"value":null}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:357)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:279)
> ... 17 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:764)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)
> ... 18 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14602) NPE when running delete_orig_table.q,update_orig_table.q under MiniLlap
[
https://issues.apache.org/jira/browse/HIVE-14602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939481#comment-15939481
]
Andy commented on HIVE-14602:
-
Is there anybody working on this problem?
Are there any updates please?
> NPE when running delete_orig_table.q,update_orig_table.q under MiniLlap
> ---
>
> Key: HIVE-14602
> URL: https://issues.apache.org/jira/browse/HIVE-14602
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>
> When delete_orig_table.q,update_orig_table.q is run under
> TestMiniLlapCliDriver, NPE is thrown
> {code}
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1471902873303_0001_31_01_00_1:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row (tag=0)
> {"key":{"reducesinkkey0":null},"value":null}
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0) {"key":{"reducesinkkey0":null},"value":null}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:289)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:279)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 15 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row (tag=0)
> {"key":{"reducesinkkey0":null},"value":null}
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:357)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:279)
> ... 17 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.get(WritableIntObjectInspector.java:36)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:764)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:348)
> ... 18 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-15644) Collect LLAP's JVM metrics via Hive's JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-15644: Assignee: Prasanth Jayachandran (was: Wei Zheng) > Collect LLAP's JVM metrics via Hive's JvmPauseMonitor > - > > Key: HIVE-15644 > URL: https://issues.apache.org/jira/browse/HIVE-15644 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Prasanth Jayachandran > Labels: llap > Attachments: HIVE-15644.1.patch > > > Similar to what Hadoop's JvmMetrics is doing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15644) Collect LLAP's JVM metrics via Hive's JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939490#comment-15939490 ] Prasanth Jayachandran commented on HIVE-15644: -- Tested this patch and I am seeing the metrics again now. Looks good to me, +1 > Collect LLAP's JVM metrics via Hive's JvmPauseMonitor > - > > Key: HIVE-15644 > URL: https://issues.apache.org/jira/browse/HIVE-15644 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Wei Zheng > Labels: llap > Attachments: HIVE-15644.1.patch > > > Similar to what Hadoop's JvmMetrics is doing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15644) Collect LLAP's JVM metrics via Hive's JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-15644: - Status: Patch Available (was: Open) > Collect LLAP's JVM metrics via Hive's JvmPauseMonitor > - > > Key: HIVE-15644 > URL: https://issues.apache.org/jira/browse/HIVE-15644 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Prasanth Jayachandran > Labels: llap > Attachments: HIVE-15644.1.patch > > > Similar to what Hadoop's JvmMetrics is doing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-15644) Collect LLAP's JVM metrics via Hive's JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-15644: Assignee: Wei Zheng > Collect LLAP's JVM metrics via Hive's JvmPauseMonitor > - > > Key: HIVE-15644 > URL: https://issues.apache.org/jira/browse/HIVE-15644 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: llap > Attachments: HIVE-15644.1.patch > > > Similar to what Hadoop's JvmMetrics is doing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-15644) Collect LLAP's JVM metrics via Hive's JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-15644: Assignee: Wei Zheng (was: Prasanth Jayachandran) > Collect LLAP's JVM metrics via Hive's JvmPauseMonitor > - > > Key: HIVE-15644 > URL: https://issues.apache.org/jira/browse/HIVE-15644 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: llap > Attachments: HIVE-15644.1.patch > > > Similar to what Hadoop's JvmMetrics is doing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16285) Servlet for dynamically configuring log levels
[ https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16285: - Attachment: HIVE-16285.2.patch > Servlet for dynamically configuring log levels > -- > > Key: HIVE-16285 > URL: https://issues.apache.org/jira/browse/HIVE-16285 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16285.1.patch, HIVE-16285.2.patch > > > Many long running services like HS2, LLAP etc. will benefit from having an > endpoint to dynamically change log levels for various loggers. This will help > greatly with debuggability without requiring a restart of the service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16286) Log canceled query id
[
https://issues.apache.org/jira/browse/HIVE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939469#comment-15939469
]
Hive QA commented on HIVE-16286:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860224/HIVE-16286.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10511 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency]
(batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=141)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4324/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4324/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4324/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860224 - PreCommit-HIVE-Build
> Log canceled query id
> -
>
> Key: HIVE-16286
> URL: https://issues.apache.org/jira/browse/HIVE-16286
> Project: Hive
> Issue Type: Improvement
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Trivial
> Attachments: HIVE-16286.1.patch
>
>
> Currently, just a generic message is logged when a query is canceled. It is
> better to log the query id as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15396) Basic Stats are not collected when for managed tables with LOCATION specified
[
https://issues.apache.org/jira/browse/HIVE-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939454#comment-15939454
]
Sahil Takiar commented on HIVE-15396:
-
[~pxiong] can't we take the location, create a {{FileSystem}} object, and the
run {{fs.exists()}} - if the location exists, then don't setup stats, if it
doesn't exist then setup full stats.
There is no guarantee that other process don't write data into the the
location, but then again there is no guarantee that other processes don't write
into {{hive.metastore.warehouse.dir}}
> Basic Stats are not collected when for managed tables with LOCATION specified
> -
>
> Key: HIVE-15396
> URL: https://issues.apache.org/jira/browse/HIVE-15396
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15396.1.patch
>
>
> Basic stats are not collected when a managed table is created with a
> specified {{LOCATION}} clause.
> {code}
> 0: jdbc:hive2://localhost:1> create table hdfs_1 (col int);
> 0: jdbc:hive2://localhost:1> describe formatted hdfs_1;
> +---++-+
> | col_name| data_type
>| comment |
> +---++-+
> | # col_name| data_type
>| comment |
> | | NULL
>| NULL|
> | col | int
>| |
> | | NULL
>| NULL|
> | # Detailed Table Information | NULL
>| NULL|
> | Database: | default
>| NULL|
> | Owner:| anonymous
>| NULL|
> | CreateTime: | Wed Mar 22 18:09:19 PDT 2017
>| NULL|
> | LastAccessTime: | UNKNOWN
>| NULL|
> | Retention:| 0
>| NULL|
> | Location: | file:/warehouse/hdfs_1 | NULL
> |
> | Table Type: | MANAGED_TABLE
>| NULL|
> | Table Parameters: | NULL
>| NULL|
> | | COLUMN_STATS_ACCURATE
>| {\"BASIC_STATS\":\"true\"} |
> | | numFiles
>| 0 |
> | | numRows
>| 0 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 0 |
> | | transient_lastDdlTime
>| 1490231359 |
> | | NULL
>| NULL|
> | # Storage Information | NULL
>| NULL|
> | SerDe Library:|
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL
>|
> | InputFormat: | org.apache.hadoop.mapred.TextInputFormat
>| NULL|
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
>|
> | Compressed: | No
>| NULL|
> | Num Buckets: | -1
>| NULL|
> | Bucket Columns: | []
>| NULL|
> |
[jira] [Commented] (HIVE-15841) Upgrade Hive to ORC 1.3.3
[ https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939452#comment-15939452 ] Prasanth Jayachandran commented on HIVE-15841: -- .2 patch with spark golden files update > Upgrade Hive to ORC 1.3.3 > - > > Key: HIVE-15841 > URL: https://issues.apache.org/jira/browse/HIVE-15841 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-15841.2.patch, HIVE-15841.patch, HIVE-15841.patch, > HIVE-15841.patch > > > Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.3 once it > releases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15880) Allow insert overwrite query to use auto.purge table property
[
https://issues.apache.org/jira/browse/HIVE-15880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-15880:
---
Attachment: HIVE-15880.02.patch
Attaching the second version of the patch which includes test cases
> Allow insert overwrite query to use auto.purge table property
> -
>
> Key: HIVE-15880
> URL: https://issues.apache.org/jira/browse/HIVE-15880
> Project: Hive
> Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-15880.01.patch, HIVE-15880.02.patch
>
>
> It seems inconsistent that auto.purge property is not considered when we do a
> INSERT OVERWRITE while it is when we do a DROP TABLE
> Drop table doesn't move table data to Trash when auto.purge is set to true
> {noformat}
> > create table temp(col1 string, col2 string);
> No rows affected (0.064 seconds)
> > alter table temp set tblproperties('auto.purge'='true');
> No rows affected (0.083 seconds)
> > insert into temp values ('test', 'test'), ('test2', 'test2');
> No rows affected (25.473 seconds)
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 22 2017-02-09 13:03
> /user/hive/warehouse/temp/00_0
> #
> > drop table temp;
> No rows affected (0.242 seconds)
> # hdfs dfs -ls /user/hive/warehouse/temp
> ls: `/user/hive/warehouse/temp': No such file or directory
> #
> # sudo -u hive hdfs dfs -ls /user/hive/.Trash/Current/user/hive/warehouse
> #
> {noformat}
> INSERT OVERWRITE query moves the table data to Trash even when auto.purge is
> set to true
> {noformat}
> > create table temp(col1 string, col2 string);
> > alter table temp set tblproperties('auto.purge'='true');
> > insert into temp values ('test', 'test'), ('test2', 'test2');
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 22 2017-02-09 13:07
> /user/hive/warehouse/temp/00_0
> #
> > insert overwrite table temp select * from dummy;
> # hdfs dfs -ls /user/hive/warehouse/temp
> Found 1 items
> -rwxrwxrwt 3 hive hive 26 2017-02-09 13:08
> /user/hive/warehouse/temp/00_0
> # sudo -u hive hdfs dfs -ls /user/hive/.Trash/Current/user/hive/warehouse
> Found 1 items
> drwx-- - hive hive 0 2017-02-09 13:08
> /user/hive/.Trash/Current/user/hive/warehouse/temp
> #
> {noformat}
> While move operations are not very costly on HDFS it could be significant
> overhead on slow FileSystems like S3. This could improve the performance of
> {{INSERT OVERWRITE TABLE}} queries especially when there are large number of
> partitions on tables located on S3 should the user wish to set auto.purge
> property to true
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-15841) Upgrade Hive to ORC 1.3.3
[ https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-15841: Assignee: Owen O'Malley (was: Prasanth Jayachandran) > Upgrade Hive to ORC 1.3.3 > - > > Key: HIVE-15841 > URL: https://issues.apache.org/jira/browse/HIVE-15841 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-15841.2.patch, HIVE-15841.patch, HIVE-15841.patch, > HIVE-15841.patch > > > Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.3 once it > releases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15841) Upgrade Hive to ORC 1.3.3
[ https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-15841: - Attachment: HIVE-15841.2.patch > Upgrade Hive to ORC 1.3.3 > - > > Key: HIVE-15841 > URL: https://issues.apache.org/jira/browse/HIVE-15841 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Prasanth Jayachandran > Attachments: HIVE-15841.2.patch, HIVE-15841.patch, HIVE-15841.patch, > HIVE-15841.patch > > > Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.3 once it > releases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-15841) Upgrade Hive to ORC 1.3.3
[ https://issues.apache.org/jira/browse/HIVE-15841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-15841: Assignee: Prasanth Jayachandran (was: Owen O'Malley) > Upgrade Hive to ORC 1.3.3 > - > > Key: HIVE-15841 > URL: https://issues.apache.org/jira/browse/HIVE-15841 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Prasanth Jayachandran > Attachments: HIVE-15841.patch, HIVE-15841.patch, HIVE-15841.patch > > > Hive needs ORC-141 and ORC-135, so we should upgrade to ORC 1.3.3 once it > releases. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15396) Basic Stats are not collected when for managed tables with LOCATION specified
[
https://issues.apache.org/jira/browse/HIVE-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939414#comment-15939414
]
Pengcheng Xiong commented on HIVE-15396:
[~stakiar], the problem is then how do u guarantee that the location is empty?
How about the other File systems?
> Basic Stats are not collected when for managed tables with LOCATION specified
> -
>
> Key: HIVE-15396
> URL: https://issues.apache.org/jira/browse/HIVE-15396
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15396.1.patch
>
>
> Basic stats are not collected when a managed table is created with a
> specified {{LOCATION}} clause.
> {code}
> 0: jdbc:hive2://localhost:1> create table hdfs_1 (col int);
> 0: jdbc:hive2://localhost:1> describe formatted hdfs_1;
> +---++-+
> | col_name| data_type
>| comment |
> +---++-+
> | # col_name| data_type
>| comment |
> | | NULL
>| NULL|
> | col | int
>| |
> | | NULL
>| NULL|
> | # Detailed Table Information | NULL
>| NULL|
> | Database: | default
>| NULL|
> | Owner:| anonymous
>| NULL|
> | CreateTime: | Wed Mar 22 18:09:19 PDT 2017
>| NULL|
> | LastAccessTime: | UNKNOWN
>| NULL|
> | Retention:| 0
>| NULL|
> | Location: | file:/warehouse/hdfs_1 | NULL
> |
> | Table Type: | MANAGED_TABLE
>| NULL|
> | Table Parameters: | NULL
>| NULL|
> | | COLUMN_STATS_ACCURATE
>| {\"BASIC_STATS\":\"true\"} |
> | | numFiles
>| 0 |
> | | numRows
>| 0 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 0 |
> | | transient_lastDdlTime
>| 1490231359 |
> | | NULL
>| NULL|
> | # Storage Information | NULL
>| NULL|
> | SerDe Library:|
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL
>|
> | InputFormat: | org.apache.hadoop.mapred.TextInputFormat
>| NULL|
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
>|
> | Compressed: | No
>| NULL|
> | Num Buckets: | -1
>| NULL|
> | Bucket Columns: | []
>| NULL|
> | Sort Columns: | []
>| NULL|
> | Storage Desc Params: | NULL
>| NULL|
> |
[jira] [Commented] (HIVE-16286) Log canceled query id
[
https://issues.apache.org/jira/browse/HIVE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939406#comment-15939406
]
Hive QA commented on HIVE-16286:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860224/HIVE-16286.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10511 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4323/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4323/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4323/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860224 - PreCommit-HIVE-Build
> Log canceled query id
> -
>
> Key: HIVE-16286
> URL: https://issues.apache.org/jira/browse/HIVE-16286
> Project: Hive
> Issue Type: Improvement
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
>Priority: Trivial
> Attachments: HIVE-16286.1.patch
>
>
> Currently, just a generic message is logged when a query is canceled. It is
> better to log the query id as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15396) Basic Stats are not collected when for managed tables with LOCATION specified
[
https://issues.apache.org/jira/browse/HIVE-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939402#comment-15939402
]
Sahil Takiar commented on HIVE-15396:
-
Thanks [~pxiong] for taking a look! I notice this behavior even when the
specified location is empty. What if I updated the patch so all stats are
collected only if the target location is empty? The use case is when running
Hive-on-S3. It's common practice to create managed Hive tables with a specified
location - e.g. {{CREATE TABLE s3_table (col int) LOCATION
's3a://[bucket-name]/s3_table/'}}
> Basic Stats are not collected when for managed tables with LOCATION specified
> -
>
> Key: HIVE-15396
> URL: https://issues.apache.org/jira/browse/HIVE-15396
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-15396.1.patch
>
>
> Basic stats are not collected when a managed table is created with a
> specified {{LOCATION}} clause.
> {code}
> 0: jdbc:hive2://localhost:1> create table hdfs_1 (col int);
> 0: jdbc:hive2://localhost:1> describe formatted hdfs_1;
> +---++-+
> | col_name| data_type
>| comment |
> +---++-+
> | # col_name| data_type
>| comment |
> | | NULL
>| NULL|
> | col | int
>| |
> | | NULL
>| NULL|
> | # Detailed Table Information | NULL
>| NULL|
> | Database: | default
>| NULL|
> | Owner:| anonymous
>| NULL|
> | CreateTime: | Wed Mar 22 18:09:19 PDT 2017
>| NULL|
> | LastAccessTime: | UNKNOWN
>| NULL|
> | Retention:| 0
>| NULL|
> | Location: | file:/warehouse/hdfs_1 | NULL
> |
> | Table Type: | MANAGED_TABLE
>| NULL|
> | Table Parameters: | NULL
>| NULL|
> | | COLUMN_STATS_ACCURATE
>| {\"BASIC_STATS\":\"true\"} |
> | | numFiles
>| 0 |
> | | numRows
>| 0 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 0 |
> | | transient_lastDdlTime
>| 1490231359 |
> | | NULL
>| NULL|
> | # Storage Information | NULL
>| NULL|
> | SerDe Library:|
> org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | NULL
>|
> | InputFormat: | org.apache.hadoop.mapred.TextInputFormat
>| NULL|
> | OutputFormat: |
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat | NULL
>|
> | Compressed: | No
>| NULL|
> | Num Buckets: | -1
>| NULL|
> | Bucket Columns: | []
>| NULL
[jira] [Commented] (HIVE-16276) Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
[ https://issues.apache.org/jira/browse/HIVE-16276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939393#comment-15939393 ] Sahil Takiar commented on HIVE-16276: - Test failures look unrelated. TestCliDriver.comments is a flaky tests - HIVE-16256 > Fix NoSuchMethodError: > com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V > -- > > Key: HIVE-16276 > URL: https://issues.apache.org/jira/browse/HIVE-16276 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-16276.1.patch, HIVE-16276.2.patch, > HIVE-16276.3.patch, HIVE-16276.4.patch, HIVE-16276.5.patch > > > The druid-handler is pulling in some com.amazonaws dependencies that conflict > with the version that Hadoop is using. This causes the above exception to be > thrown when running Hive against S3. This patch fixes the dependency issue by > shading the aws dependencies in the druid artifacts. Unfortunately, I can't > find a great way to add a test for this so it doesn't happen in the future. > We will need some more robust S3-integration tests for that. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16289) add hints for semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-16289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16289: Status: Patch Available (was: Open) > add hints for semijoin reduction > > > Key: HIVE-16289 > URL: https://issues.apache.org/jira/browse/HIVE-16289 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16289.patch > > > For now hints will only impact bloom filter size if semijoin is enabled. > In a follow-up, after some cost-based semi-join decision logic is added, they > may also influence it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15644) Collect LLAP's JVM metrics via Hive's JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15644: - Attachment: HIVE-15644.1.patch > Collect LLAP's JVM metrics via Hive's JvmPauseMonitor > - > > Key: HIVE-15644 > URL: https://issues.apache.org/jira/browse/HIVE-15644 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Wei Zheng > Labels: llap > Attachments: HIVE-15644.1.patch > > > Similar to what Hadoop's JvmMetrics is doing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15644) Collect LLAP's JVM metrics via Hive's JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15644: - Summary: Collect LLAP's JVM metrics via Hive's JvmPauseMonitor (was: Collect JVM metrics via JvmPauseMonitor) > Collect LLAP's JVM metrics via Hive's JvmPauseMonitor > - > > Key: HIVE-15644 > URL: https://issues.apache.org/jira/browse/HIVE-15644 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Wei Zheng > Labels: llap > > Similar to what Hadoop's JvmMetrics is doing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15644) Collect JVM metrics via JvmPauseMonitor
[ https://issues.apache.org/jira/browse/HIVE-15644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-15644: - Labels: llap (was: ) > Collect JVM metrics via JvmPauseMonitor > --- > > Key: HIVE-15644 > URL: https://issues.apache.org/jira/browse/HIVE-15644 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.2.0 >Reporter: Wei Zheng > Labels: llap > > Similar to what Hadoop's JvmMetrics is doing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15211) Provide support for complex expressions in ON clauses for INNER joins
[
https://issues.apache.org/jira/browse/HIVE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939351#comment-15939351
]
Caleb Jones commented on HIVE-15211:
Ah, sorry, I meant I hit this with Hive < 2.2.0. I have not yet tried with
2.2.0.
> Provide support for complex expressions in ON clauses for INNER joins
> -
>
> Key: HIVE-15211
> URL: https://issues.apache.org/jira/browse/HIVE-15211
> Project: Hive
> Issue Type: Bug
> Components: CBO, Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-15211.01.patch, HIVE-15211.patch
>
>
> Currently, we have some restrictions on the predicates that we can use in ON
> clauses for inner joins (we have those restrictions for outer joins too, but
> we will tackle that in a follow-up). Semantically equivalent queries can be
> expressed if the predicate is introduced in the WHERE clause, but we would
> like that user can express it both in ON and WHERE clause, as in standard SQL.
> This patch is an extension to overcome these restrictions for inner joins.
> It will allow to write queries that currently fail in Hive such as:
> {code:sql}
> -- Disjunctions
> SELECT *
> FROM src1 JOIN src
> ON (src1.key=src.key
> OR src1.value between 100 and 102
> OR src.value between 100 and 102)
> LIMIT 10;
> -- Conjunction with multiple inputs references in one side
> SELECT *
> FROM src1 JOIN src
> ON (src1.key+src.key >= 100
> AND src1.key+src.key <= 102)
> LIMIT 10;
> -- Conjunct with no references
> SELECT *
> FROM src1 JOIN src
> ON (src1.value between 100 and 102
> AND src.value between 100 and 102
> AND true)
> LIMIT 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16289) add hints for semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-16289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16289: Attachment: HIVE-16289.patch [~djaiswal] [~jdere] can you take a look? > add hints for semijoin reduction > > > Key: HIVE-16289 > URL: https://issues.apache.org/jira/browse/HIVE-16289 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-16289.patch > > > For now hints will only impact bloom filter size if semijoin is enabled. > In a follow-up, after some cost-based semi-join decision logic is added, they > may also influence it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16219) metastore notification_log contains serialized message with non functional fields
[
https://issues.apache.org/jira/browse/HIVE-16219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939321#comment-15939321
]
Vaibhav Gumashta commented on HIVE-16219:
-
+1
> metastore notification_log contains serialized message with non functional
> fields
> --
>
> Key: HIVE-16219
> URL: https://issues.apache.org/jira/browse/HIVE-16219
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.2.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 2.2.0
>
> Attachments: HIVE-16219.3.patch
>
>
> the event notification logs stored in hive metastore have json serialized
> messages stored in NOTIFICATION_LOG table, these messages also store the
> serialized Thrift API objects in them. when doing a reply dump we are however
> serializing both the metadata for replication event + event Message +
> additional helper method getters representing the thrift objects.
> We should only serialize metadata for replication event + event Message
> for ex for create table :
> {code}
> {
> "eventType": "CREATE_TABLE",
> "server": "",
> "servicePrincipal": "",
> "db": "default",
> "table": "a",
> "tableObjJson":
> "{\"1\":{\"str\":\"a\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1489552350},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"name\"},\"2\":{\"str\":\"string\"}}]},\"2\":{\"str\":\"file:/tmp/warehouse/a\"},\"3\":{\"str\":\"org.apache.hadoop.mapred.TextInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":-1},\"7\":{\"rec\":{\"2\":{\"str\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\"},\"3\":{\"map\":[\"str\",\"str\",2,{\"field.delim\":\"\\n\",\"serialization.format\":\"\\n\"}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]},\"11\":{\"rec\":{\"1\":{\"lst\":[\"str\",0]},\"2\":{\"lst\":[\"lst\",0]},\"3\":{\"map\":[\"lst\",\"str\",0,{}]}}},\"12\":{\"tf\":0}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"totalSize\":\"0\",\"EXTERNAL\":\"TRUE\",\"numRows\":\"0\",\"rawDataSize\":\"0\",\"COLUMN_STATS_ACCURATE\":\"{\\\"BASIC_STATS\\\":\\\"true\\\"}\",\"numFiles\":\"0\",\"transient_lastDdlTime\":\"1489552350\"}]},\"12\":{\"str\":\"EXTERNAL_TABLE\"},\"13\":{\"rec\":{\"1\":{\"map\":[\"str\",\"lst\",1,{\"anagarwal\":[\"rec\",4,{\"1\":{\"str\":\"INSERT\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"SELECT\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"UPDATE\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}},{\"1\":{\"str\":\"DELETE\"},\"2\":{\"i32\":-1},\"3\":{\"str\":\"anagarwal\"},\"4\":{\"i32\":1},\"5\":{\"tf\":1}}]}]}}},\"14\":{\"tf\":0}}",
> "timestamp": 1489552350,
> "files": [],
> "tableObj": {
> "tableName": "a",
> "dbName": "default",
> "owner": "anagarwal",
> "createTime": 1489552350,
> "lastAccessTime": 0,
> "retention": 0,
> "sd": {
> "cols": [
> {
> "name": "name",
> "type": "string",
> "comment": null,
> "setName": true,
> "setType": true,
> "setComment": false
> }
> ],
> "location": "file:/tmp/warehouse/a",
> "inputFormat": "org.apache.hadoop.mapred.TextInputFormat",
> "outputFormat":
> "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
> "compressed": false,
> "numBuckets": -1,
> "serdeInfo": {
> "name": null,
> "serializationLib":
> "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe",
> "parameters": {
> "serialization.format": "\n",
> "field.delim": "\n"
> },
> "setName": false,
> "parametersSize": 2,
> "setParameters": true,
> "setSerializationLib": true
> },
> "bucketCols": [],
> "sortCols": [],
> "parameters": {},
> "skewedInfo": {
> "skewedColNames": [],
> "skewedColValues": [],
> "skewedColValueLocationMaps": {},
> "setSkewedColNames": true,
> "setSkewedColValues": true,
> "setSkewedColValueLocationMaps": true,
> "skewedColNamesSize": 0,
> "skewedColNamesIterator": [],
> "skewedColValuesSize": 0,
> "skewedColValuesIterator": [],
> "skewedColValueLocationMapsSize": 0
> },
> "storedAsSubDirectories": false,
> "setSkewedInfo": true,
> "parametersSize": 0,
> "colsSize": 1,
> "setParameters": true,
> "setLoc
[jira] [Commented] (HIVE-16276) Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
[
https://issues.apache.org/jira/browse/HIVE-16276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939317#comment-15939317
]
Hive QA commented on HIVE-16276:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860220/HIVE-16276.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10510 tests
executed
*Failed tests:*
{noformat}
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=211)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4322/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4322/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4322/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860220 - PreCommit-HIVE-Build
> Fix NoSuchMethodError:
> com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> --
>
> Key: HIVE-16276
> URL: https://issues.apache.org/jira/browse/HIVE-16276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16276.1.patch, HIVE-16276.2.patch,
> HIVE-16276.3.patch, HIVE-16276.4.patch, HIVE-16276.5.patch
>
>
> The druid-handler is pulling in some com.amazonaws dependencies that conflict
> with the version that Hadoop is using. This causes the above exception to be
> thrown when running Hive against S3. This patch fixes the dependency issue by
> shading the aws dependencies in the druid artifacts. Unfortunately, I can't
> find a great way to add a test for this so it doesn't happen in the future.
> We will need some more robust S3-integration tests for that.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939316#comment-15939316
]
Vihang Karajgaonkar commented on HIVE-16287:
I can take a look at this ..
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar reassigned HIVE-16287:
--
Assignee: Vihang Karajgaonkar
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15211) Provide support for complex expressions in ON clauses for INNER joins
[
https://issues.apache.org/jira/browse/HIVE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939314#comment-15939314
]
Jesus Camacho Rodriguez commented on HIVE-15211:
[~calebjones], have you tried with latest master? You should not hit that
limitation anymore, as we should support any arbitrary condition. Please, let
me know if you do and the error you hit.
> Provide support for complex expressions in ON clauses for INNER joins
> -
>
> Key: HIVE-15211
> URL: https://issues.apache.org/jira/browse/HIVE-15211
> Project: Hive
> Issue Type: Bug
> Components: CBO, Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-15211.01.patch, HIVE-15211.patch
>
>
> Currently, we have some restrictions on the predicates that we can use in ON
> clauses for inner joins (we have those restrictions for outer joins too, but
> we will tackle that in a follow-up). Semantically equivalent queries can be
> expressed if the predicate is introduced in the WHERE clause, but we would
> like that user can express it both in ON and WHERE clause, as in standard SQL.
> This patch is an extension to overcome these restrictions for inner joins.
> It will allow to write queries that currently fail in Hive such as:
> {code:sql}
> -- Disjunctions
> SELECT *
> FROM src1 JOIN src
> ON (src1.key=src.key
> OR src1.value between 100 and 102
> OR src.value between 100 and 102)
> LIMIT 10;
> -- Conjunction with multiple inputs references in one side
> SELECT *
> FROM src1 JOIN src
> ON (src1.key+src.key >= 100
> AND src1.key+src.key <= 102)
> LIMIT 10;
> -- Conjunct with no references
> SELECT *
> FROM src1 JOIN src
> ON (src1.value between 100 and 102
> AND src.value between 100 and 102
> AND true)
> LIMIT 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-15211) Provide support for complex expressions in ON clauses for INNER joins
[
https://issues.apache.org/jira/browse/HIVE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939303#comment-15939303
]
Caleb Jones edited comment on HIVE-15211 at 3/23/17 10:14 PM:
--
Will UDFs be supported in complex expressions in the ON clause?
I hit this limitation when I had two tables I wanted to join on based on the
intersection of array columns.
{noformat}
create table tbl_a (
val string,
ids array
);
create table tbl_b (
val string,
ids array
);
add jar hdfs:///brickhouse-0.7.1-SNAPSHOT.jar;
select a.val, b.val
from tbl_a as a
join tbl_b as b
on (size(intersect_array(a.ids, b.ids)) > 0);
{noformat}
An intersection size > 0 is a somewhat trivial case, but if requirements were
such that the intersection size was needed within a specific range it makes
this query much more difficult.
was (Author: calebjones):
Will UDFs be supported in complex expressions in the ON clause?
I hit this limitation when I had two tables I wanted to join on based on the
intersection of array columns.
{noformat}
create table tbl_a (
val string,
ids array
);
create table tbl_b (
val string,
ids array
);
add jar hdfs:///brickhouse-0.7.1-SNAPSHOT.jar;
select a.val, b.val
from tbl_a as a
join tbl_b as b
on (size(intersect_array(a.ids, b.ids)) > 0);
{noformat}
> Provide support for complex expressions in ON clauses for INNER joins
> -
>
> Key: HIVE-15211
> URL: https://issues.apache.org/jira/browse/HIVE-15211
> Project: Hive
> Issue Type: Bug
> Components: CBO, Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-15211.01.patch, HIVE-15211.patch
>
>
> Currently, we have some restrictions on the predicates that we can use in ON
> clauses for inner joins (we have those restrictions for outer joins too, but
> we will tackle that in a follow-up). Semantically equivalent queries can be
> expressed if the predicate is introduced in the WHERE clause, but we would
> like that user can express it both in ON and WHERE clause, as in standard SQL.
> This patch is an extension to overcome these restrictions for inner joins.
> It will allow to write queries that currently fail in Hive such as:
> {code:sql}
> -- Disjunctions
> SELECT *
> FROM src1 JOIN src
> ON (src1.key=src.key
> OR src1.value between 100 and 102
> OR src.value between 100 and 102)
> LIMIT 10;
> -- Conjunction with multiple inputs references in one side
> SELECT *
> FROM src1 JOIN src
> ON (src1.key+src.key >= 100
> AND src1.key+src.key <= 102)
> LIMIT 10;
> -- Conjunct with no references
> SELECT *
> FROM src1 JOIN src
> ON (src1.value between 100 and 102
> AND src.value between 100 and 102
> AND true)
> LIMIT 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16289) add hints for semijoin reduction
[ https://issues.apache.org/jira/browse/HIVE-16289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-16289: --- > add hints for semijoin reduction > > > Key: HIVE-16289 > URL: https://issues.apache.org/jira/browse/HIVE-16289 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > For now hints will only impact bloom filter size if semijoin is enabled. > In a follow-up, after some cost-based semi-join decision logic is added, they > may also influence it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15211) Provide support for complex expressions in ON clauses for INNER joins
[
https://issues.apache.org/jira/browse/HIVE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939303#comment-15939303
]
Caleb Jones commented on HIVE-15211:
Will UDFs be supported in complex expressions in the ON clause?
I hit this limitation when I had two tables I wanted to join on based on the
intersection of array columns.
{noformat}
create table tbl_a (
val string,
ids array
);
create table tbl_b (
val string,
ids array
);
add jar hdfs:///brickhouse-0.7.1-SNAPSHOT.jar;
select a.val, b.val
from tbl_a as a
join tbl_b as b
on (size(intersect_array(a.ids, b.ids)) > 0);
{noformat}
> Provide support for complex expressions in ON clauses for INNER joins
> -
>
> Key: HIVE-15211
> URL: https://issues.apache.org/jira/browse/HIVE-15211
> Project: Hive
> Issue Type: Bug
> Components: CBO, Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Labels: TODOC2.2
> Fix For: 2.2.0
>
> Attachments: HIVE-15211.01.patch, HIVE-15211.patch
>
>
> Currently, we have some restrictions on the predicates that we can use in ON
> clauses for inner joins (we have those restrictions for outer joins too, but
> we will tackle that in a follow-up). Semantically equivalent queries can be
> expressed if the predicate is introduced in the WHERE clause, but we would
> like that user can express it both in ON and WHERE clause, as in standard SQL.
> This patch is an extension to overcome these restrictions for inner joins.
> It will allow to write queries that currently fail in Hive such as:
> {code:sql}
> -- Disjunctions
> SELECT *
> FROM src1 JOIN src
> ON (src1.key=src.key
> OR src1.value between 100 and 102
> OR src.value between 100 and 102)
> LIMIT 10;
> -- Conjunction with multiple inputs references in one side
> SELECT *
> FROM src1 JOIN src
> ON (src1.key+src.key >= 100
> AND src1.key+src.key <= 102)
> LIMIT 10;
> -- Conjunct with no references
> SELECT *
> FROM src1 JOIN src
> ON (src1.value between 100 and 102
> AND src.value between 100 and 102
> AND true)
> LIMIT 10;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16288) Add blobstore tests for ORC and RCFILE file formats
[ https://issues.apache.org/jira/browse/HIVE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Poepping updated HIVE-16288: --- Attachment: HIVE-16288.patch Attached patch > Add blobstore tests for ORC and RCFILE file formats > --- > > Key: HIVE-16288 > URL: https://issues.apache.org/jira/browse/HIVE-16288 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16288.patch > > > This patch adds four tests each for ORC and RCFILE when running against > blobstore filesystems: > * Test for bucketed tables > * Test for nonpartitioned tables > * Test for partitioned tables > * Test for partitioned tables with nonstandard partition locations -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16274) Support tuning of NDV of columns using lower/upper bounds
[ https://issues.apache.org/jira/browse/HIVE-16274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939290#comment-15939290 ] Jason Dere commented on HIVE-16274: --- Unfortunately this scales the nDV of all column in the same way, which makes the nDV look high for all columns. If the column has a min/max range, or data type limits, could we at least bound it by those? The point of having this was because the density function estimate does not work well in some cases. Longer term it might be nice to have some improvements in that area as well. > Support tuning of NDV of columns using lower/upper bounds > - > > Key: HIVE-16274 > URL: https://issues.apache.org/jira/browse/HIVE-16274 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16274.01.patch > > > For partitioned tables, the distinct value (nDV) estimate for a column is by > default set to the largest nDV value in any of the partitions being > considered, which is a lower bound on the nDV estimate. > This provides a config setting to allow the estimate to a specified fraction > (0.0 - 1.0) of the higher bound on the nDV estimate (the sum of all the nDVs > in all partitions). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16288) Add blobstore tests for ORC and RCFILE file formats
[ https://issues.apache.org/jira/browse/HIVE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Poepping updated HIVE-16288: --- Status: Patch Available (was: Open) > Add blobstore tests for ORC and RCFILE file formats > --- > > Key: HIVE-16288 > URL: https://issues.apache.org/jira/browse/HIVE-16288 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16288.patch > > > This patch adds four tests each for ORC and RCFILE when running against > blobstore filesystems: > * Test for bucketed tables > * Test for nonpartitioned tables > * Test for partitioned tables > * Test for partitioned tables with nonstandard partition locations -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16288) Add blobstore tests for ORC and RCFILE file formats
[ https://issues.apache.org/jira/browse/HIVE-16288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Poepping reassigned HIVE-16288: -- > Add blobstore tests for ORC and RCFILE file formats > --- > > Key: HIVE-16288 > URL: https://issues.apache.org/jira/browse/HIVE-16288 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > > This patch adds four tests each for ORC and RCFILE when running against > blobstore filesystems: > * Test for bucketed tables > * Test for nonpartitioned tables > * Test for partitioned tables > * Test for partitioned tables with nonstandard partition locations -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16274) Support tuning of NDV of columns using lower/upper bounds
[ https://issues.apache.org/jira/browse/HIVE-16274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-16274: -- Description: For partitioned tables, the distinct value (nDV) estimate for a column is by default set to the largest nDV value in any of the partitions being considered, which is a lower bound on the nDV estimate. This provides a config setting to allow the estimate to a specified fraction (0.0 - 1.0) of the higher bound on the nDV estimate (the sum of all the nDVs in all partitions). > Support tuning of NDV of columns using lower/upper bounds > - > > Key: HIVE-16274 > URL: https://issues.apache.org/jira/browse/HIVE-16274 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-16274.01.patch > > > For partitioned tables, the distinct value (nDV) estimate for a column is by > default set to the largest nDV value in any of the partitions being > considered, which is a lower bound on the nDV estimate. > This provides a config setting to allow the estimate to a specified fraction > (0.0 - 1.0) of the higher bound on the nDV estimate (the sum of all the nDVs > in all partitions). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-13282:
---
Target Version/s: 1.3.0 (was: 1.3.0, 2.2.0)
> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Matt McCline
>Priority: Blocker
> Attachments: HIVE-13282.01.patch, smb_fail_issue.patch,
> smb_groupby.q, smb_groupby.q.out
>
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException
> when they incorrectly initialize themselves with tag 0 but the incoming tag
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value)
> rt2) vt2
> where vt1.id=vt2.id;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939267#comment-15939267
]
Matt McCline commented on HIVE-13282:
-
Probably not.
> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Matt McCline
>Priority: Blocker
> Attachments: HIVE-13282.01.patch, smb_fail_issue.patch,
> smb_groupby.q, smb_groupby.q.out
>
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException
> when they incorrectly initialize themselves with tag 0 but the incoming tag
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value)
> rt2) vt2
> where vt1.id=vt2.id;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16285) Servlet for dynamically configuring log levels
[ https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939264#comment-15939264 ] Prasanth Jayachandran commented on HIVE-16285: -- [~gopalv]/[~sseth] can someone please take a look? > Servlet for dynamically configuring log levels > -- > > Key: HIVE-16285 > URL: https://issues.apache.org/jira/browse/HIVE-16285 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16285.1.patch > > > Many long running services like HS2, LLAP etc. will benefit from having an > endpoint to dynamically change log levels for various loggers. This will help > greatly with debuggability without requiring a restart of the service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16285) Servlet for dynamically configuring log levels
[ https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939263#comment-15939263 ] Prasanth Jayachandran commented on HIVE-16285: -- Removed all the statics that holds log level enable states. This used to be a problem with old log4j 1.x. With log4j2 calls to Log.isInfoEnabled() (also other levels) are very cheap (very few nano seconds - https://logging.apache.org/log4j/log4j-2.2/performance.html). Also branch predictors should do a good job with such conditions in inner loops. > Servlet for dynamically configuring log levels > -- > > Key: HIVE-16285 > URL: https://issues.apache.org/jira/browse/HIVE-16285 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16285.1.patch > > > Many long running services like HS2, LLAP etc. will benefit from having an > endpoint to dynamically change log levels for various loggers. This will help > greatly with debuggability without requiring a restart of the service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16285) Servlet for dynamically configuring log levels
[ https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16285: - Status: Patch Available (was: Open) > Servlet for dynamically configuring log levels > -- > > Key: HIVE-16285 > URL: https://issues.apache.org/jira/browse/HIVE-16285 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16285.1.patch > > > Many long running services like HS2, LLAP etc. will benefit from having an > endpoint to dynamically change log levels for various loggers. This will help > greatly with debuggability without requiring a restart of the service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16285) Servlet for dynamically configuring log levels
[ https://issues.apache.org/jira/browse/HIVE-16285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16285: - Attachment: HIVE-16285.1.patch > Servlet for dynamically configuring log levels > -- > > Key: HIVE-16285 > URL: https://issues.apache.org/jira/browse/HIVE-16285 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.2.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-16285.1.patch > > > Many long running services like HS2, LLAP etc. will benefit from having an > endpoint to dynamically change log levels for various loggers. This will help > greatly with debuggability without requiring a restart of the service. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16152) TestBeeLineDriver logging improvements
[ https://issues.apache.org/jira/browse/HIVE-16152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939259#comment-15939259 ] Yongzhi Chen commented on HIVE-16152: - PATCH 2 looks good. +1 > TestBeeLineDriver logging improvements > -- > > Key: HIVE-16152 > URL: https://issues.apache.org/jira/browse/HIVE-16152 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16152.2.patch, HIVE-16152.patch > > > During the review of HIVE-16127 we agreed, that it would be great to have > improved logging and error messages during the TestBeeLineDriver run. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16154) Determine when dynamic runtime filtering should be disabled
[ https://issues.apache.org/jira/browse/HIVE-16154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-16154: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master > Determine when dynamic runtime filtering should be disabled > --- > > Key: HIVE-16154 > URL: https://issues.apache.org/jira/browse/HIVE-16154 > Project: Hive > Issue Type: Bug > Components: Query Planning >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: 2.2.0 > > Attachments: HIVE-16154.1.patch, HIVE-16154.2.patch, > HIVE-16154.3.patch, HIVE-16154.4.patch > > > Currently dynamic min/max/bloom optimization is always enabled. However there > are times where it may not be beneficial, such as if the semijoin has a PK-FK > relation and there are no filters on the semijoin table. Try to devise a way > to do a cost/benefit calculation to see if there is enough benefit to adding > the runtime filter. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13370) Add test for HIVE-11470
[ https://issues.apache.org/jira/browse/HIVE-13370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939247#comment-15939247 ] Daniel Dai commented on HIVE-13370: --- Yes, this is not committed. There is no reason leave it idle. I triggered a precommit test #4325, will commit once test pass. > Add test for HIVE-11470 > --- > > Key: HIVE-13370 > URL: https://issues.apache.org/jira/browse/HIVE-13370 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Blocker > Attachments: HIVE-13370.patch > > > HIVE-11470 added capability to handle NULL dynamic partitioning keys > properly. However, it did not add a test for the case, we should have one so > we don't have future regressions of the same. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15665) LLAP: OrcFileMetadata objects in cache can impact heap usage
[
https://issues.apache.org/jira/browse/HIVE-15665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939248#comment-15939248
]
Hive QA commented on HIVE-15665:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860214/HIVE-15665.04.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 9890 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=55)
[load_nonpart_authsuccess.q,udf_ceiling.q,llap_uncompressed.q,join_cond_pushdown_2.q,parquet_mixed_partition_formats2.q,udf_count.q,ctas_colname.q,dbtxnmgr_query3.q,parquet_ppd_boolean.q,index_auto_empty.q,index_in_db.q,vectorization_1.q,stats_ppr_all.q,script_env_var2.q,msck_repair_2.q,auto_join_filters.q,join0.q,udf_xpath_int.q,ba_table1.q,groupby3_map_skew.q,vector_udf1.q,join14_hadoop20.q,udf_map.q,druid_timeseries.q,exim_16_part_external.q,merge3.q,input35.q,auto_smb_mapjoin_14.q,create_table_like_stats.q,join37.q]
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=56)
[input31.q,list_bucket_query_oneskew_1.q,database_drop.q,union19.q,udf_rlike.q,multi_insert_union_src.q,union24.q,implicit_cast1.q,varchar_cast.q,input_columnarserde.q,index_compact_binary_search.q,pcr.q,avro_comments.q,alter_rename_partition_authorization.q,mergejoin.q,groupby4_noskew.q,alter_merge_stats.q,vector_complex_all.q,bucketsortoptimize_insert_1.q,orc_empty_strings.q,union_remove_5.q,groupby7_noskew_multi_single_reducer.q,input9.q,llap_udf.q,filter_cond_pushdown.q,skewjoin_mapjoin6.q,index_auto_self_join.q,authorization_view_2.q,input33.q,reduce_deduplicate_extended2.q]
TestOrcMetadataCache - did not produce a TEST-*.xml file (likely timed out)
(batchId=281)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_3]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[combine3] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[empty_join] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[escape_crlf] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_numeric]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_bitmap_compression]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part1] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join38] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_query_multiskew_3]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_reader] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[masking_8] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_min_max] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order2] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ppd_partition]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_decode_name]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_wise_fileformat11]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_wise_fileformat6]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[print_header] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats4] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf9] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_bitmap_and]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_minute] (batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_4]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_non_partitioned]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_1]
(batchId=74)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
(batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
(batchId=136)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
(batchId=137)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
(batchId=138)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapL
[jira] [Commented] (HIVE-16276) Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
[ https://issues.apache.org/jira/browse/HIVE-16276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939221#comment-15939221 ] Jesus Camacho Rodriguez commented on HIVE-16276: +1 > Fix NoSuchMethodError: > com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V > -- > > Key: HIVE-16276 > URL: https://issues.apache.org/jira/browse/HIVE-16276 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-16276.1.patch, HIVE-16276.2.patch, > HIVE-16276.3.patch, HIVE-16276.4.patch, HIVE-16276.5.patch > > > The druid-handler is pulling in some com.amazonaws dependencies that conflict > with the version that Hadoop is using. This causes the above exception to be > thrown when running Hive against S3. This patch fixes the dependency issue by > shading the aws dependencies in the druid artifacts. Unfortunately, I can't > find a great way to add a test for this so it doesn't happen in the future. > We will need some more robust S3-integration tests for that. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14077) revert or fix HIVE-13380
[ https://issues.apache.org/jira/browse/HIVE-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939211#comment-15939211 ] Sergey Shelukhin commented on HIVE-14077: - We reverted it from 2.1, because it breaks TPCDS/H and also ANSI compliance. Some reexamination is necessary, either it should be reverted or fixed. By now I don't remember if there's reason to not revert it, except for b/c (of note is that it was not released in 2.1 so I am not sure what the concern with that was). > revert or fix HIVE-13380 > > > Key: HIVE-14077 > URL: https://issues.apache.org/jira/browse/HIVE-14077 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Blocker > > See comments in that JIRA -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15829) LLAP text cache: disable memory tracking on the writer
[ https://issues.apache.org/jira/browse/HIVE-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939208#comment-15939208 ] Sergey Shelukhin commented on HIVE-15829: - It should. [~prasanth_j] is working on ORC upgrade I think. After that, we can finish this. > LLAP text cache: disable memory tracking on the writer > -- > > Key: HIVE-15829 > URL: https://issues.apache.org/jira/browse/HIVE-15829 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > > See ORC-141 and HIVE-15672 for context -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15923) Hive default partition causes errors in get partitions
[
https://issues.apache.org/jira/browse/HIVE-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939204#comment-15939204
]
Sergey Shelukhin commented on HIVE-15923:
-
Yes, should be a blocker. I will look at it later today/this week
> Hive default partition causes errors in get partitions
> --
>
> Key: HIVE-15923
> URL: https://issues.apache.org/jira/browse/HIVE-15923
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15923.01.patch, HIVE-15923.patch
>
>
> This is the ORM error, direct SQL fails too before that, with a similar error.
> {noformat}
> 2017-02-14T17:45:11,158 ERROR [09fdd887-0164-4f55-97e9-4ba147d962be main]
> metastore.ObjectStore:java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDefaultDesc cannot be cast to
> java.lang.Long
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaLongObjectInspector.get(JavaLongObjectInspector.java:40)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getDouble(PrimitiveObjectInspectorUtils.java:801)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DoubleConverter.convert(P
> rimitiveObjectInspectorConverter.java:240)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqualOrGreaterThan.evaluate(GenericUDFOPEqualOrGreaterThan.java:145)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween.evaluate(GenericUDFBetween.java:57)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:88)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:63)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.evaluateExprOnPart(PartExprEvalUtils.java:126)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-13282) GroupBy and select operator encounter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939200#comment-15939200
]
Pengcheng Xiong commented on HIVE-13282:
[~mmccline], will this go into 2.2.0? Thanks
> GroupBy and select operator encounter ArrayIndexOutOfBoundsException
>
>
> Key: HIVE-13282
> URL: https://issues.apache.org/jira/browse/HIVE-13282
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 1.2.1, 2.0.0, 2.1.0
>Reporter: Vikram Dixit K
>Assignee: Matt McCline
>Priority: Blocker
> Attachments: HIVE-13282.01.patch, smb_fail_issue.patch,
> smb_groupby.q, smb_groupby.q.out
>
>
> The group by and select operators run into the ArrayIndexOutOfBoundsException
> when they incorrectly initialize themselves with tag 0 but the incoming tag
> id is different.
> {code}
> select count(*) from
> (select rt1.id from
> (select t1.key as id, t1.value as od from tab t1 group by key, value) rt1) vt1
> join
> (select rt2.id from
> (select t2.key as id, t2.value as od from tab_part t2 group by key, value)
> rt2) vt2
> where vt1.id=vt2.id;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-14077) revert or fix HIVE-13380
[ https://issues.apache.org/jira/browse/HIVE-14077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939198#comment-15939198 ] Pengcheng Xiong commented on HIVE-14077: [~sershe] and [~xuefuz], will this go into 2.2.0? Thanks > revert or fix HIVE-13380 > > > Key: HIVE-14077 > URL: https://issues.apache.org/jira/browse/HIVE-14077 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Priority: Blocker > > See comments in that JIRA -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15829) LLAP text cache: disable memory tracking on the writer
[ https://issues.apache.org/jira/browse/HIVE-15829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939197#comment-15939197 ] Pengcheng Xiong commented on HIVE-15829: will this go into 2.2.0? Thanks. > LLAP text cache: disable memory tracking on the writer > -- > > Key: HIVE-15829 > URL: https://issues.apache.org/jira/browse/HIVE-15829 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > > See ORC-141 and HIVE-15672 for context -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13370) Add test for HIVE-11470
[ https://issues.apache.org/jira/browse/HIVE-13370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939196#comment-15939196 ] Pengcheng Xiong commented on HIVE-13370: [~sushanth], i could not find this in master. will this go into 2.2.0? Thanks. > Add test for HIVE-11470 > --- > > Key: HIVE-13370 > URL: https://issues.apache.org/jira/browse/HIVE-13370 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan >Priority: Blocker > Attachments: HIVE-13370.patch > > > HIVE-11470 added capability to handle NULL dynamic partitioning keys > properly. However, it did not add a test for the case, we should have one so > we don't have future regressions of the same. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15923) Hive default partition causes errors in get partitions
[
https://issues.apache.org/jira/browse/HIVE-15923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939193#comment-15939193
]
Pengcheng Xiong commented on HIVE-15923:
[~sershe] will this go into 2.2.0? Thanks.
> Hive default partition causes errors in get partitions
> --
>
> Key: HIVE-15923
> URL: https://issues.apache.org/jira/browse/HIVE-15923
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: HIVE-15923.01.patch, HIVE-15923.patch
>
>
> This is the ORM error, direct SQL fails too before that, with a similar error.
> {noformat}
> 2017-02-14T17:45:11,158 ERROR [09fdd887-0164-4f55-97e9-4ba147d962be main]
> metastore.ObjectStore:java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.plan.ExprNodeConstantDefaultDesc cannot be cast to
> java.lang.Long
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaLongObjectInspector.get(JavaLongObjectInspector.java:40)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils.getDouble(PrimitiveObjectInspectorUtils.java:801)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]at
> org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorConverter$DoubleConverter.convert(P
> rimitiveObjectInspectorConverter.java:240)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqualOrGreaterThan.evaluate(GenericUDFOPEqualOrGreaterThan.java:145)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFBetween.evaluate(GenericUDFBetween.java:57)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:88)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPAnd.evaluate(GenericUDFOPAnd.java:63)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:187)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:80)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:68)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.optimizer.ppr.PartExprEvalUtils.evaluateExprOnPart(PartExprEvalUtils.java:126)
> ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16286) Log canceled query id
[ https://issues.apache.org/jira/browse/HIVE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-16286: --- Attachment: HIVE-16286.1.patch > Log canceled query id > - > > Key: HIVE-16286 > URL: https://issues.apache.org/jira/browse/HIVE-16286 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Attachments: HIVE-16286.1.patch > > > Currently, just a generic message is logged when a query is canceled. It is > better to log the query id as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ying Chen updated HIVE-16287:
-
Description:
I was renaming my partition in a table that I've created using the location
clause, and noticed that when after rename is completed, my partition is moved
to the hive warehouse (hive.metastore.warehouse.dir).
{quote}
create table test_local_part (col1 int) partitioned by (col2 int) location
'/tmp/testtable/test_local_part';
insert into test_local_part partition (col2=1) values (1),(3);
insert into test_local_part partition (col2=2) values (3);
alter table test_local_part partition (col2='1') rename to partition (col2='4');
{quote}
Running:
describe formatted test_local_part partition (col2='2')
# Detailed Partition Information
Partition Value:[2]
Database: default
Table: test_local_part
CreateTime: Mon Mar 20 13:25:28 PDT 2017
LastAccessTime: UNKNOWN
Protect Mode: None
Location:
*hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
Running:
describe formatted test_local_part partition (col2='4')
# Detailed Partition Information
Partition Value:[4]
Database: default
Table: test_local_part
CreateTime: Mon Mar 20 13:24:53 PDT 2017
LastAccessTime: UNKNOWN
Protect Mode: None
Location:
*hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
---
Per Sergio's comment - "The rename should create the new partition name in the
same location of the table. "
was:
I was renaming my partition in a table that I've created using the location
clause, and noticed that when after rename is completed, my partition is moved
to the hive warehouse (hive.metastore.warehouse.dir).
{quote}
create table test_local_part (col1 int) partitioned by (col2 int) location
'/tmp/testtable/test_local_part';
insert into test_local_part partition (col2=1) values (1),(3);
insert into test_local_part partition (col2=2) values (3);
alter table test_local_part partition (col2='1') rename to partition (col2='4');
{quote}
Running:
{{describe formatted test_local_part partition (col2='2')}}
# Detailed Partition Information
Partition Value:[2]
Database: default
Table: test_local_part
CreateTime: Mon Mar 20 13:25:28 PDT 2017
LastAccessTime: UNKNOWN
Protect Mode: None
Location:
*hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
Running:
{{describe formatted test_local_part partition (col2='4')}}
# Detailed Partition Information
Partition Value:[4]
Database: default
Table: test_local_part
CreateTime: Mon Mar 20 13:24:53 PDT 2017
LastAccessTime: UNKNOWN
Protect Mode: None
Location:
*hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Priority: Minor
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
[jira] [Assigned] (HIVE-16286) Log canceled query id
[ https://issues.apache.org/jira/browse/HIVE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang reassigned HIVE-16286: -- > Log canceled query id > - > > Key: HIVE-16286 > URL: https://issues.apache.org/jira/browse/HIVE-16286 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > > Currently, just a generic message is logged when a query is canceled. It is > better to log the query id as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16286) Log canceled query id
[ https://issues.apache.org/jira/browse/HIVE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939188#comment-15939188 ] Xuefu Zhang commented on HIVE-16286: +1 > Log canceled query id > - > > Key: HIVE-16286 > URL: https://issues.apache.org/jira/browse/HIVE-16286 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Attachments: HIVE-16286.1.patch > > > Currently, just a generic message is logged when a query is canceled. It is > better to log the query id as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16286) Log canceled query id
[ https://issues.apache.org/jira/browse/HIVE-16286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-16286: --- Status: Patch Available (was: Open) > Log canceled query id > - > > Key: HIVE-16286 > URL: https://issues.apache.org/jira/browse/HIVE-16286 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Attachments: HIVE-16286.1.patch > > > Currently, just a generic message is logged when a query is canceled. It is > better to log the query id as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16276) Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
[ https://issues.apache.org/jira/browse/HIVE-16276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939183#comment-15939183 ] slim bouguerra commented on HIVE-16276: --- LGTM > Fix NoSuchMethodError: > com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V > -- > > Key: HIVE-16276 > URL: https://issues.apache.org/jira/browse/HIVE-16276 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-16276.1.patch, HIVE-16276.2.patch, > HIVE-16276.3.patch, HIVE-16276.4.patch, HIVE-16276.5.patch > > > The druid-handler is pulling in some com.amazonaws dependencies that conflict > with the version that Hadoop is using. This causes the above exception to be > thrown when running Hive against S3. This patch fixes the dependency issue by > shading the aws dependencies in the druid artifacts. Unfortunately, I can't > find a great way to add a test for this so it doesn't happen in the future. > We will need some more robust S3-integration tests for that. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16276) Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
[ https://issues.apache.org/jira/browse/HIVE-16276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-16276: Attachment: HIVE-16276.5.patch Thanks [~jcamachorodriguez] and [~bslim]. I've attached an updated patch, and validated locally that Hive-on-S3 works. Does the patch look good? > Fix NoSuchMethodError: > com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V > -- > > Key: HIVE-16276 > URL: https://issues.apache.org/jira/browse/HIVE-16276 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Attachments: HIVE-16276.1.patch, HIVE-16276.2.patch, > HIVE-16276.3.patch, HIVE-16276.4.patch, HIVE-16276.5.patch > > > The druid-handler is pulling in some com.amazonaws dependencies that conflict > with the version that Hadoop is using. This causes the above exception to be > thrown when running Hive against S3. This patch fixes the dependency issue by > shading the aws dependencies in the druid artifacts. Unfortunately, I can't > find a great way to add a test for this so it doesn't happen in the future. > We will need some more robust S3-integration tests for that. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16222) add a setting to disable row.serde for specific formats; enable for others
[
https://issues.apache.org/jira/browse/HIVE-16222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939169#comment-15939169
]
Hive QA commented on HIVE-16222:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860208/HIVE-16222.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10507 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=233)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4320/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4320/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4320/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860208 - PreCommit-HIVE-Build
> add a setting to disable row.serde for specific formats; enable for others
> --
>
> Key: HIVE-16222
> URL: https://issues.apache.org/jira/browse/HIVE-16222
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16222.01.patch, HIVE-16222.02.patch,
> HIVE-16222.03.patch, HIVE-16222.patch
>
>
> Per [~gopalv]
> {quote}
> row.serde = true ... breaks Parquet (they expect to get the same object back,
> which means you can't buffer 1024 rows).
> {quote}
> We want to enable this and vector.serde for text vectorization. Need to turn
> it off for specific formats.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16206) Make Codahale metrics reporters pluggable
[ https://issues.apache.org/jira/browse/HIVE-16206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939098#comment-15939098 ] Ratandeep Ratti commented on HIVE-16206: The changes look good to me [~sbeeram]! > Make Codahale metrics reporters pluggable > - > > Key: HIVE-16206 > URL: https://issues.apache.org/jira/browse/HIVE-16206 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 2.1.2 >Reporter: Sunitha Beeram >Assignee: Sunitha Beeram > Attachments: HIVE-16206.2.patch, HIVE-16206.3.patch, > HIVE-16206.4.patch, HIVE-16206.5.patch, HIVE-16206.6.patch, HIVE-16206.patch > > > Hive metrics code currently allows pluggable metrics handlers - ie, handlers > that take care of providing interfaces for metrics collection as well as a > reporting; one of the 'handlers' is CodahaleMetrics. Codahale can work with > different reporters - currently supported ones are Console, JMX, JSON file > and hadoop2 sink. However, adding a new reporter involves changing that > class. We would like to make this conf driven just the way MetricsFactory > handles configurable Metrics classes. > Scope of work: > - Provide a new configuration option, HIVE_CODAHALE_REPORTER_CLASSES that > enumerates classes (like HIVE_METRICS_CLASS and unlike HIVE_METRICS_REPORTER). > - Move JsonFileReporter into its own class. > - Update CodahaleMetrics.java to read new config option and if the new option > is not present, look for the old option and instantiate accordingly) - ie, > make the code backward compatible. > - Update and add new tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16239) remove useless hiveserver
[
https://issues.apache.org/jira/browse/HIVE-16239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939083#comment-15939083
]
Vaibhav Gumashta commented on HIVE-16239:
-
+1
Thanks [~ferhui]
> remove useless hiveserver
> -
>
> Key: HIVE-16239
> URL: https://issues.apache.org/jira/browse/HIVE-16239
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 2.0.1, 2.1.1
>Reporter: Fei Hui
>Assignee: Fei Hui
> Attachments: HIVE-16239.1-branch-2.0.patch,
> HIVE-16239.1-branch-2.1.patch
>
>
> {quote}
> [hadoop@header hive]$ hive --service hiveserver
> Starting Hive Thrift Server
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/apps/apache-hive-2.0.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/apps/spark-1.6.2-bin-hadoop2.7/lib/spark-assembly-1.6.2-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/apps/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Exception in thread "main" java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.service.HiveServer
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:348)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {quote}
> hiveserver does not exist, we should remove hiveserver from cli on branch-2.0
> After removing it, we get useful message
> {quote}
> Service hiveserver not found
> Available Services: beeline cli hbaseimport hbaseschematool help
> hiveburninclient hiveserver2 hplsql hwi jar lineage llap metastore metatool
> orcfiledump rcfilecat schemaTool version
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15665) LLAP: OrcFileMetadata objects in cache can impact heap usage
[ https://issues.apache.org/jira/browse/HIVE-15665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-15665: Attachment: HIVE-15665.04.patch Another rebase... HiveQA failed to post the failure to apply the patch. > LLAP: OrcFileMetadata objects in cache can impact heap usage > > > Key: HIVE-15665 > URL: https://issues.apache.org/jira/browse/HIVE-15665 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Rajesh Balamohan >Assignee: Sergey Shelukhin > Attachments: HIVE-15665.01.patch, HIVE-15665.02.patch, > HIVE-15665.03.patch, HIVE-15665.04.patch, HIVE-15665.patch > > > OrcFileMetadata internally has filestats, stripestats etc which are allocated > in heap. On large data sets, this could have an impact on the heap usage and > the memory usage by different executors in LLAP. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[
https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939048#comment-15939048
]
Hive QA commented on HIVE-13517:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12860189/HIVE-13517.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10511 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4319/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4319/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4319/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12860189 - PreCommit-HIVE-Build
> Hive logs in Spark Executor and Driver should show thread-id.
> -
>
> Key: HIVE-13517
> URL: https://issues.apache.org/jira/browse/HIVE-13517
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1, 2.0.0
>Reporter: Szehon Ho
>Assignee: Sahil Takiar
> Attachments: executor-driver-log.PNG, HIVE-13517.1.patch,
> HIVE-13517.2.patch
>
>
> In Spark, there might be more than one task running in one executor.
> Similarly, there may be more than one thread running in Driver.
> This makes debugging through the logs a nightmare. It would be great if there
> could be thread-ids in the logs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16278) LLAP: metadata cache may incorrectly decrease memory usage in mem manager
[ https://issues.apache.org/jira/browse/HIVE-16278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16278: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > LLAP: metadata cache may incorrectly decrease memory usage in mem manager > - > > Key: HIVE-16278 > URL: https://issues.apache.org/jira/browse/HIVE-16278 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.2.0 > > Attachments: HIVE-16278.01.patch, HIVE-16278.02.patch, > HIVE-16278.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16276) Fix NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
[
https://issues.apache.org/jira/browse/HIVE-16276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939019#comment-15939019
]
slim bouguerra commented on HIVE-16276:
---
yes we can exclude it.
{code}
diff --git druid-handler/pom.xml druid-handler/pom.xml
index d536589..d19f07b 100644
--- druid-handler/pom.xml
+++ druid-handler/pom.xml
@@ -107,6 +107,10 @@
com.fasterxml.jackson.core
jackson-databind
+
+ io.druid
+ druid-aws-common
+
{code}
> Fix NoSuchMethodError:
> com.amazonaws.services.s3.transfer.TransferManagerConfiguration.setMultipartUploadThreshold(I)V
> --
>
> Key: HIVE-16276
> URL: https://issues.apache.org/jira/browse/HIVE-16276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16276.1.patch, HIVE-16276.2.patch,
> HIVE-16276.3.patch, HIVE-16276.4.patch
>
>
> The druid-handler is pulling in some com.amazonaws dependencies that conflict
> with the version that Hadoop is using. This causes the above exception to be
> thrown when running Hive against S3. This patch fixes the dependency issue by
> shading the aws dependencies in the druid artifacts. Unfortunately, I can't
> find a great way to add a test for this so it doesn't happen in the future.
> We will need some more robust S3-integration tests for that.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16222) add a setting to disable row.serde for specific formats; enable for others
[
https://issues.apache.org/jira/browse/HIVE-16222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-16222:
Attachment: HIVE-16222.03.patch
Updated the test outputs.
> add a setting to disable row.serde for specific formats; enable for others
> --
>
> Key: HIVE-16222
> URL: https://issues.apache.org/jira/browse/HIVE-16222
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-16222.01.patch, HIVE-16222.02.patch,
> HIVE-16222.03.patch, HIVE-16222.patch
>
>
> Per [~gopalv]
> {quote}
> row.serde = true ... breaks Parquet (they expect to get the same object back,
> which means you can't buffer 1024 rows).
> {quote}
> We want to enable this and vector.serde for text vectorization. Need to turn
> it off for specific formats.
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
