[jira] [Commented] (HIVE-15025) Secure-Socket-Layer (SSL) support for HMS
[
https://issues.apache.org/jira/browse/HIVE-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604186#comment-15604186
]
Hive QA commented on HIVE-15025:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12835007/HIVE-15025.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1778/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1778/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1778/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException:
org.apache.hive.ptest.execution.ssh.SSHExecutionException: RSyncResult
[localFile=/data/hiveptest/logs/PreCommit-HIVE-Build-1778/succeeded/98-TestSparkCliDriver-avro_joins.q-skewjoinopt16.q-auto_join14.q-and-12-more,
remoteFile=/home/hiveptest/104.154.176.70-hiveptest-0/logs/,
getExitCode()=255, getException()=null, getUser()=hiveptest,
getHost()=104.154.176.70, getInstance()=0]: 'ssh: connect to host
104.154.176.70 port 22: Connection refused
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.154.176.70 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.154.176.70 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.154.176.70 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.154.176.70 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12835007 - PreCommit-HIVE-Build
> Secure-Socket-Layer (SSL) support for HMS
> -
>
> Key: HIVE-15025
> URL: https://issues.apache.org/jira/browse/HIVE-15025
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-15025.1.patch, HIVE-15025.2.patch,
> HIVE-15025.3.patch
>
>
> HMS server should support SSL encryption. When the server is keberos enabled,
> the encryption can be enabled. But if keberos is not enabled, then there is
> no encryption between HS2 and HMS.
> Similar to HS2, we should support encryption in both cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14887) Reduce the memory requirements for tests
[
https://issues.apache.org/jira/browse/HIVE-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604081#comment-15604081
]
Hive QA commented on HIVE-14887:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12834714/HIVE-14887.06.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1777/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1777/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1777/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Tests exited with: ExecutionException: java.util.concurrent.ExecutionException:
org.apache.hive.ptest.execution.ssh.SSHExecutionException: RSyncResult
[localFile=/data/hiveptest/logs/PreCommit-HIVE-Build-1777/succeeded/128-TestSparkCliDriver-union_remove_15.q-bucket_map_join_tez1.q-groupby7_noskew.q-and-12-more,
remoteFile=/home/hiveptest/104.154.176.70-hiveptest-1/logs/,
getExitCode()=255, getException()=null, getUser()=hiveptest,
getHost()=104.154.176.70, getInstance()=1]: 'ssh: connect to host
104.154.176.70 port 22: Connection refused
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.154.176.70 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.154.176.70 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.154.176.70 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
ssh: connect to host 104.154.176.70 port 22: Connection timed out
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12834714 - PreCommit-HIVE-Build
> Reduce the memory requirements for tests
>
>
> Key: HIVE-14887
> URL: https://issues.apache.org/jira/browse/HIVE-14887
> Project: Hive
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14887.01.patch, HIVE-14887.02.patch,
> HIVE-14887.03.patch, HIVE-14887.04.patch, HIVE-14887.05.patch,
> HIVE-14887.06.patch
>
>
> The clusters that we spin up end up requiring 16GB at times. Also the maven
> arguments seem a little heavy weight.
> Reducing this will allow for additional ptest drones per box, which should
> bring down the runtime.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13589) beeline support prompt for password with '-p' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-13589: Resolution: Fixed Status: Resolved (was: Patch Available) Failed cases are not related. Committed to the master. Thanks [~vihangk1] and [~Jk_Self] for the contributions and reviews. > beeline support prompt for password with '-p' option > > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Vihang Karajgaonkar > Fix For: 2.2.0 > > Attachments: HIVE-13589.1.patch, HIVE-13589.10.patch, > HIVE-13589.11.patch, HIVE-13589.12.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, > HIVE-13589.6.patch, HIVE-13589.7.patch, HIVE-13589.8.patch, HIVE-13589.9.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Attachment: HIVE-14909.2.patch I am not able to reproduce the test failure of TestSemanticAnalysis.testAlterTableRename in my local machine. Create a patch with more debug to see what happened. > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.2.patch, HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14964) Failing Test: Fix TestBeelineArgParsing tests
[
https://issues.apache.org/jira/browse/HIVE-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15604016#comment-15604016
]
Hive QA commented on HIVE-14964:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12834696/HIVE-14964.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10593 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_bulk]
(batchId=89)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
(batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=164)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1776/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1776/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1776/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12834696 - PreCommit-HIVE-Build
> Failing Test: Fix TestBeelineArgParsing tests
> -
>
> Key: HIVE-14964
> URL: https://issues.apache.org/jira/browse/HIVE-14964
> Project: Hive
> Issue Type: Sub-task
> Components: Tests
>Reporter: Jason Dere
>Assignee: Zoltan Haindrich
> Attachments: HIVE-14964.1.patch
>
>
> Failing last several builds:
> {noformat}
> org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] 0.12
> sec12
>
> org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
> 29 ms 12
> org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] 42 ms
> 12
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13873) Column pruning for nested fields
[ https://issues.apache.org/jira/browse/HIVE-13873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-13873: Attachment: HIVE-13873.6.patch > Column pruning for nested fields > > > Key: HIVE-13873 > URL: https://issues.apache.org/jira/browse/HIVE-13873 > Project: Hive > Issue Type: New Feature > Components: Logical Optimizer >Reporter: Xuefu Zhang >Assignee: Ferdinand Xu > Attachments: HIVE-13873.1.patch, HIVE-13873.2.patch, > HIVE-13873.3.patch, HIVE-13873.4.patch, HIVE-13873.5.patch, > HIVE-13873.6.patch, HIVE-13873.patch, HIVE-13873.wip.patch > > > Some columnar file formats such as Parquet store fields in struct type also > column by column using encoding described in Google Dramel pager. It's very > common in big data where data are stored in structs while queries only needs > a subset of the the fields in the structs. However, presently Hive still > needs to read the whole struct regardless whether all fields are selected. > Therefore, pruning unwanted sub-fields in struct or nested fields at file > reading time would be a big performance boost for such scenarios. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10901) Optimize mutli column distinct queries
[
https://issues.apache.org/jira/browse/HIVE-10901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603939#comment-15603939
]
Hive QA commented on HIVE-10901:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12737058/HIVE-10901.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1775/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1775/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1775/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-10-25 02:26:04.338
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-1775/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-10-25 02:26:04.341
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
080de97..7968e1e master -> origin/master
+ git reset --hard HEAD
HEAD is now at 080de97 HIVE-14950 Support integer data type (Zoltan Haindrich
via Alan Gates)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 7968e1e HIVE-14837: JDBC: standalone jar is missing hadoop core
dependencies (Tao Li, via Gopal V)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-10-25 02:26:05.441
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveExpandDistinctAggregatesRule.java:
No such file or directory
error: a/ql/src/test/results/clientpositive/tez/limit_pushdown.q.out: No such
file or directory
error: a/ql/src/test/results/clientpositive/tez/mrr.q.out: No such file or
directory
error: a/ql/src/test/results/clientpositive/tez/vectorization_limit.q.out: No
such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12737058 - PreCommit-HIVE-Build
> Optimize mutli column distinct queries
>
>
> Key: HIVE-10901
> URL: https://issues.apache.org/jira/browse/HIVE-10901
> Project: Hive
> Issue Type: New Feature
> Components: CBO, Logical Optimizer
>Affects Versions: 1.2.0
>Reporter: Mostafa Mokhtar
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-10901.patch
>
>
> HIVE-10568 is useful only when there is a distinct on one column. It can be
> expanded for multiple column cases too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive
[
https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603935#comment-15603935
]
Hive QA commented on HIVE-14909:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12835004/HIVE-14909.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10592 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
(batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=164)
org.apache.hive.hcatalog.cli.TestSemanticAnalysis.testAlterTableRename
(batchId=172)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1774/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1774/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1774/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12835004 - PreCommit-HIVE-Build
> Preserve the "parent location" of the table when an "alter table
> rename to " is submitted (the case when the db location is not
> specified and the Hive defult db is outside the same encrypted zone).
> --
>
> Key: HIVE-14909
> URL: https://issues.apache.org/jira/browse/HIVE-14909
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 1.1.0
>Reporter: Adriano
>Assignee: Chaoyu Tang
> Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch,
> HIVE-14909.patch, HIVE-14909.patch
>
>
> Alter Table operation for db_enc.rename_test failed to move data due to:
> '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption
> zone.'
> When Hive renames a managed table, it always creates the new renamed table
> directory under its database directory in order to keep a db/table hierarchy.
> In this case, the renamed table directory is created under "default db"
> directory "hive/warehouse/". When Hive renames a managed table, it always
> creates the new renamed table directory under its database directory in order
> to keep a db/table hierarchy. In this case, the renamed table directory is
> created under "default' db directory typically set as /hive/warehouse/ .
> This error doesn't appear if first create a database which points to a
> directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't
> have this problem. For example,
> create database db_enc location '/hdfs/encrypted_path/db_enc;
> use db_enc;
> create table rename_test (...) location
> '/hdfs/encrypted_path/db_enc/rename_test';
> alter table rename_test rename to test_rename;
> The renamed test_rename directory is created under
> /hdfs/encrypted_path/db_enc.
> Considering that the encryption of a filesystem is part of the evolution
> hardening of a system (where the system and the data contained can already
> exists) and a db can be already created without location set (because it is
> not strictly required)and the default db is outside the same encryption zone
> (or in a no-encryption zone) the alter table rename operation will fail.
> Improvement:
> Preserve the "parent location" of the table when an "alter table
> rename to " is submitted (the case when the db location is not
> specified and the Hive defult db is outside the same encrypted zone).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14837) JDBC: standalone jar is missing hadoop core dependencies
[
https://issues.apache.org/jira/browse/HIVE-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-14837:
---
Resolution: Fixed
Fix Version/s: 2.2.0
Release Note: JDBC: standalone jar is missing hadoop core dependencies
(Tao Li, via Gopal V)
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~taoli-hwx] & [~rajesh.balamohan].
> JDBC: standalone jar is missing hadoop core dependencies
>
>
> Key: HIVE-14837
> URL: https://issues.apache.org/jira/browse/HIVE-14837
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Fix For: 2.2.0
>
> Attachments: HIVE-14837.1.patch
>
>
> {code}
> 2016/09/24 00:31:57 ERROR - jmeter.threads.JMeterThread: Test failed!
> java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
> at
> org.apache.hive.jdbc.HiveConnection.createUnderlyingTransport(HiveConnection.java:418)
> at
> org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:438)
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:225)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14837) JDBC: standalone jar is missing hadoop core dependencies
[
https://issues.apache.org/jira/browse/HIVE-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603781#comment-15603781
]
Hive QA commented on HIVE-14837:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12834051/HIVE-14837.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10592 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
(batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=164)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1773/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1773/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1773/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12834051 - PreCommit-HIVE-Build
> JDBC: standalone jar is missing hadoop core dependencies
>
>
> Key: HIVE-14837
> URL: https://issues.apache.org/jira/browse/HIVE-14837
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Tao Li
> Attachments: HIVE-14837.1.patch
>
>
> {code}
> 2016/09/24 00:31:57 ERROR - jmeter.threads.JMeterThread: Test failed!
> java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
> at
> org.apache.hive.jdbc.HiveConnection.createUnderlyingTransport(HiveConnection.java:418)
> at
> org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:438)
> at
> org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:225)
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13995) Hive metastore queries when there is no filter on partition column are inefficient
[
https://issues.apache.org/jira/browse/HIVE-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603606#comment-15603606
]
Hive QA commented on HIVE-13995:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12819476/HIVE-13995.7.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1772/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1772/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1772/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-10-24 23:45:30.033
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-1772/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-10-24 23:45:30.036
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 080de97 HIVE-14950 Support integer data type (Zoltan Haindrich
via Alan Gates)
+ git clean -f -d
Removing
itests/hive-unit/src/test/java/org/apache/hive/beeline/TestBeelinePasswordOption.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 080de97 HIVE-14950 Support integer data type (Zoltan Haindrich
via Alan Gates)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-10-24 23:45:30.940
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
No such file or directory
error:
a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/hbase/TestHBaseAggrStatsCacheIntegration.java:
No such file or directory
error: a/metastore/if/hive_metastore.thrift: No such file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp: No such
file or directory
error: a/metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h: No such file
or directory
error:
a/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PartitionsStatsRequest.java:
No such file or directory
error: a/metastore/src/gen/thrift/gen-php/metastore/Types.php: No such file or
directory
error: a/metastore/src/gen/thrift/gen-py/hive_metastore/ttypes.py: No such file
or directory
error: a/metastore/src/gen/thrift/gen-rb/hive_metastore_types.rb: No such file
or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/AggregateStatsCache.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java:
No such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
No such file or directory
error: a/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java: No
such file or directory
error:
a/metastore/src/java/org/apache/hadoop/hive/metastore/hbase/HBaseStore.java: No
such file or directory
error:
a/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java:
No such file or directory
error:
[jira] [Commented] (HIVE-13589) beeline support prompt for password with '-p' option
[
https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603599#comment-15603599
]
Hive QA commented on HIVE-13589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12834984/HIVE-13589.12.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10602 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
(batchId=216)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
(batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=164)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1771/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1771/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1771/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12834984 - PreCommit-HIVE-Build
> beeline support prompt for password with '-p' option
>
>
> Key: HIVE-13589
> URL: https://issues.apache.org/jira/browse/HIVE-13589
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Reporter: Thejas M Nair
>Assignee: Vihang Karajgaonkar
> Fix For: 2.2.0
>
> Attachments: HIVE-13589.1.patch, HIVE-13589.10.patch,
> HIVE-13589.11.patch, HIVE-13589.12.patch, HIVE-13589.2.patch,
> HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch,
> HIVE-13589.6.patch, HIVE-13589.7.patch, HIVE-13589.8.patch, HIVE-13589.9.patch
>
>
> Specifying connection string using commandline options in beeline is
> convenient, as it gets saved in shell command history, and it is easy to
> retrieve it from there.
> However, specifying the password in command prompt is not secure as it gets
> displayed on screen and saved in the history.
> It should be possible to specify '-p' without an argument to make beeline
> prompt for password.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-15047) metastore setconf API can set too many confs
[
https://issues.apache.org/jira/browse/HIVE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin resolved HIVE-15047.
-
Resolution: Invalid
Nm, I looked at the wrong piece of code. There's metaVars, metaConfVars, and
metaConfs
> metastore setconf API can set too many confs
>
>
> Key: HIVE-15047
> URL: https://issues.apache.org/jira/browse/HIVE-15047
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> {noformat}
> public void setMetaConf(String key, String value) throws MetaException {
> ConfVars confVar = HiveConf.getMetaConf(key);
> {noformat}
> that gets stuff from "metaConfs", which contains all the metastore configs,
> including the warehouse location, port (setting that unfortunately would not
> work; imagine the possibilities!), etc.
> There's another small list near metaConfs - metaConfVars, billed as "User
> configurable Metastore vars". That is probably the list that was intended for
> the above API; as it is, it's not used for any checks.
> When making the change SetProcessor etc. should be adjusted to fail with some
> user-friendly message when trying to set unsettable metastore configs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15047) metastore setconf API can set too many confs
[
https://issues.apache.org/jira/browse/HIVE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-15047:
Description:
{noformat}
public void setMetaConf(String key, String value) throws MetaException {
ConfVars confVar = HiveConf.getMetaConf(key);
{noformat}
that gets stuff from "metaConfs", which contains all the metastore configs,
including the warehouse location, port (setting that unfortunately would not
work; imagine the possibilities!), etc.
There's another small list near metaConfs - metaConfVars, billed as "User
configurable Metastore vars". That is probably the list that was intended for
the above API; as it is, it's not used for any checks.
When making the change SetProcessor etc. should be adjusted to fail with some
user-friendly message when trying to set unsettable metastore configs.
was:
{noformat}
public void setMetaConf(String key, String value) throws MetaException {
ConfVars confVar = HiveConf.getMetaConf(key);
{noformat}
that gets stuff from "metaConfs", which contains all the metastore configs,
including the warehouse location, port (setting that unfortunately would not
work; imagine the possibilities!), etc.
There's another small list near metaConfs - metaConfVars, billed as "User
configurable Metastore vars". That is probably the list that should be used; as
it is, it's not used for any checks.
When making the change SetProcessor etc. should be adjusted to fail with some
user-friendly message when trying to set unsettable metastore APIs.
{noformat}
> metastore setconf API can set too many confs
>
>
> Key: HIVE-15047
> URL: https://issues.apache.org/jira/browse/HIVE-15047
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> {noformat}
> public void setMetaConf(String key, String value) throws MetaException {
> ConfVars confVar = HiveConf.getMetaConf(key);
> {noformat}
> that gets stuff from "metaConfs", which contains all the metastore configs,
> including the warehouse location, port (setting that unfortunately would not
> work; imagine the possibilities!), etc.
> There's another small list near metaConfs - metaConfVars, billed as "User
> configurable Metastore vars". That is probably the list that was intended for
> the above API; as it is, it's not used for any checks.
> When making the change SetProcessor etc. should be adjusted to fail with some
> user-friendly message when trying to set unsettable metastore configs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15047) metastore setconf API can set too many confs
[
https://issues.apache.org/jira/browse/HIVE-15047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603554#comment-15603554
]
Sergey Shelukhin commented on HIVE-15047:
-
[~sushanth] [~thejas] fyi
> metastore setconf API can set too many confs
>
>
> Key: HIVE-15047
> URL: https://issues.apache.org/jira/browse/HIVE-15047
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> {noformat}
> public void setMetaConf(String key, String value) throws MetaException {
> ConfVars confVar = HiveConf.getMetaConf(key);
> {noformat}
> that gets stuff from "metaConfs", which contains all the metastore configs,
> including the warehouse location, port (setting that unfortunately would not
> work; imagine the possibilities!), etc.
> There's another small list near metaConfs - metaConfVars, billed as "User
> configurable Metastore vars". That is probably the list that should be used;
> as it is, it's not used for any checks.
> When making the change SetProcessor etc. should be adjusted to fail with some
> user-friendly message when trying to set unsettable metastore APIs.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14646) poison metastore APIs to make sure we can fail old clients for backward compat
[ https://issues.apache.org/jira/browse/HIVE-14646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-14646: --- Assignee: Sergey Shelukhin > poison metastore APIs to make sure we can fail old clients for backward compat > -- > > Key: HIVE-14646 > URL: https://issues.apache.org/jira/browse/HIVE-14646 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14950) Support integer data type
[ https://issues.apache.org/jira/browse/HIVE-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603497#comment-15603497 ] Alan Gates commented on HIVE-14950: --- [~leftylev], I updated https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL noting that INTEGER will now be a keyword in HIVE 2.2.0. Is there anywhere I need to change? > Support integer data type > - > > Key: HIVE-14950 > URL: https://issues.apache.org/jira/browse/HIVE-14950 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14950.1.patch, HIVE-14950.2.patch > > > maybe its just me bumping into this difference again and again... > but it's in the sql2011 standard... > adding an alias for int would be easy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14582) Add trunc(numeric) udf
[ https://issues.apache.org/jira/browse/HIVE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603492#comment-15603492 ] Ashutosh Chauhan commented on HIVE-14582: - [~chinnalalam] Are you working on this? > Add trunc(numeric) udf > -- > > Key: HIVE-14582 > URL: https://issues.apache.org/jira/browse/HIVE-14582 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Ashutosh Chauhan >Assignee: Chinna Rao Lalam > Attachments: HIVE-14582.patch > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions200.htm -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14950) Support integer data type
[ https://issues.apache.org/jira/browse/HIVE-14950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-14950: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed patch to master. Thanks Zoltan. > Support integer data type > - > > Key: HIVE-14950 > URL: https://issues.apache.org/jira/browse/HIVE-14950 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14950.1.patch, HIVE-14950.2.patch > > > maybe its just me bumping into this difference again and again... > but it's in the sql2011 standard... > adding an alias for int would be easy. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15029) Add logic to estimate stats for BETWEEN operator
[ https://issues.apache.org/jira/browse/HIVE-15029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603480#comment-15603480 ] Ashutosh Chauhan commented on HIVE-15029: - +1 > Add logic to estimate stats for BETWEEN operator > > > Key: HIVE-15029 > URL: https://issues.apache.org/jira/browse/HIVE-15029 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-15029.01.patch, HIVE-15029.02.patch > > > Currently, BETWEEN operator is considered in the default case: reduces the > input rows to the half. This may lead to wrong estimates for the number of > rows produced by Filter operators. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15030) Fixes in inference of collation for Tez cost model
[ https://issues.apache.org/jira/browse/HIVE-15030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603470#comment-15603470 ] Ashutosh Chauhan commented on HIVE-15030: - +1 > Fixes in inference of collation for Tez cost model > -- > > Key: HIVE-15030 > URL: https://issues.apache.org/jira/browse/HIVE-15030 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-15030.patch > > > Tez cost model might get NPE if collation returned by join algorithm is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15042) Support intersect/except without distinct keyword
[
https://issues.apache.org/jira/browse/HIVE-15042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603464#comment-15603464
]
Hive QA commented on HIVE-15042:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12834977/HIVE-15042.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10583 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
(batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=164)
org.apache.hive.service.cli.session.TestSessionManagerMetrics.testAbandonedSessionMetrics
(batchId=183)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1770/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1770/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1770/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12834977 - PreCommit-HIVE-Build
> Support intersect/except without distinct keyword
> -
>
> Key: HIVE-15042
> URL: https://issues.apache.org/jira/browse/HIVE-15042
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-15042.01.patch
>
>
> basically, intersect = intersect distinct.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14821) build q test
[ https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14821: -- Attachment: (was: HIVE-14821.4.patch) > build q test > > > Key: HIVE-14821 > URL: https://issues.apache.org/jira/browse/HIVE-14821 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14821) build q test
[ https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14821: -- Attachment: (was: HIVE-14821.3.patch) > build q test > > > Key: HIVE-14821 > URL: https://issues.apache.org/jira/browse/HIVE-14821 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14821) build q test
[ https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14821: -- Attachment: (was: HIVE-14821.1.patch) > build q test > > > Key: HIVE-14821 > URL: https://issues.apache.org/jira/browse/HIVE-14821 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14821) build q test
[ https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14821: -- Attachment: (was: HIVE-14821.2.patch) > build q test > > > Key: HIVE-14821 > URL: https://issues.apache.org/jira/browse/HIVE-14821 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14821) build q test
[ https://issues.apache.org/jira/browse/HIVE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14821: -- Attachment: (was: HIVE-14821.patch) > build q test > > > Key: HIVE-14821 > URL: https://issues.apache.org/jira/browse/HIVE-14821 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14855) test patch
[ https://issues.apache.org/jira/browse/HIVE-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14855: -- Attachment: (was: HIVE-14855.5.patch) > test patch > -- > > Key: HIVE-14855 > URL: https://issues.apache.org/jira/browse/HIVE-14855 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14855) test patch
[ https://issues.apache.org/jira/browse/HIVE-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14855: -- Attachment: (was: HIVE-14855.3.patch) > test patch > -- > > Key: HIVE-14855 > URL: https://issues.apache.org/jira/browse/HIVE-14855 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14855) test patch
[ https://issues.apache.org/jira/browse/HIVE-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14855: -- Attachment: (was: HIVE-14855.2.patch) > test patch > -- > > Key: HIVE-14855 > URL: https://issues.apache.org/jira/browse/HIVE-14855 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14855) test patch
[ https://issues.apache.org/jira/browse/HIVE-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14855: -- Attachment: (was: HIVE-14855.patch) > test patch > -- > > Key: HIVE-14855 > URL: https://issues.apache.org/jira/browse/HIVE-14855 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15029) Add logic to estimate stats for BETWEEN operator
[ https://issues.apache.org/jira/browse/HIVE-15029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-15029: --- Attachment: HIVE-15029.02.patch [~ashutoshc], I have updated the q file changes in the patch. Could you take a look? Thanks! > Add logic to estimate stats for BETWEEN operator > > > Key: HIVE-15029 > URL: https://issues.apache.org/jira/browse/HIVE-15029 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-15029.01.patch, HIVE-15029.02.patch > > > Currently, BETWEEN operator is considered in the default case: reduces the > input rows to the half. This may lead to wrong estimates for the number of > rows produced by Filter operators. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14855) test patch
[ https://issues.apache.org/jira/browse/HIVE-14855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14855: -- Attachment: (was: HIVE-14855.4.patch) > test patch > -- > > Key: HIVE-14855 > URL: https://issues.apache.org/jira/browse/HIVE-14855 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15046) Druid query type not recognized after Calcite upgrade
[ https://issues.apache.org/jira/browse/HIVE-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15046: Status: Patch Available (was: Open) +1 > Druid query type not recognized after Calcite upgrade > - > > Key: HIVE-15046 > URL: https://issues.apache.org/jira/browse/HIVE-15046 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-15046.patch > > > Introduced by HIVE-13316. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14992) Relocate several common libraries in hive jdbc uber jar
[ https://issues.apache.org/jira/browse/HIVE-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603359#comment-15603359 ] Tao Li commented on HIVE-14992: --- Test failures are not related. > Relocate several common libraries in hive jdbc uber jar > --- > > Key: HIVE-14992 > URL: https://issues.apache.org/jira/browse/HIVE-14992 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14992.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14391) TestAccumuloCliDriver is not executed during precommit tests
[
https://issues.apache.org/jira/browse/HIVE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated HIVE-14391:
--
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
> TestAccumuloCliDriver is not executed during precommit tests
>
>
> Key: HIVE-14391
> URL: https://issues.apache.org/jira/browse/HIVE-14391
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Peter Vary
> Fix For: 2.2.0
>
> Attachments: HIVE-14391.2.patch, HIVE-14391.patch
>
>
> according to for example this build result:
> https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/685/testReport/org.apache.hadoop.hive.cli/
> there is no 'TestAccumuloCliDriver' being run during precommit testing...but
> i see no reason why and how it was excluded inside the project;
> my maven executes it when i start it with {{-Dtest=TestAccumuloCliDriver}} -
> so i think the properties/profiles aren't preventing it.
> maybe i miss something obvious ;)
> (note: my TestAccumuloCliDriver executions are failed with errors.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14391) TestAccumuloCliDriver is not executed during precommit tests
[
https://issues.apache.org/jira/browse/HIVE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603326#comment-15603326
]
Siddharth Seth commented on HIVE-14391:
---
If this is being used, I'm going to go ahead and commit the jira. We can open
another one to discuss maintenance of this handler for future major releases.
Thanks for the patch [~pvary]
> TestAccumuloCliDriver is not executed during precommit tests
>
>
> Key: HIVE-14391
> URL: https://issues.apache.org/jira/browse/HIVE-14391
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Peter Vary
> Attachments: HIVE-14391.2.patch, HIVE-14391.patch
>
>
> according to for example this build result:
> https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/685/testReport/org.apache.hadoop.hive.cli/
> there is no 'TestAccumuloCliDriver' being run during precommit testing...but
> i see no reason why and how it was excluded inside the project;
> my maven executes it when i start it with {{-Dtest=TestAccumuloCliDriver}} -
> so i think the properties/profiles aren't preventing it.
> maybe i miss something obvious ;)
> (note: my TestAccumuloCliDriver executions are failed with errors.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14496) Enable Calcite rewriting with materialized views
[ https://issues.apache.org/jira/browse/HIVE-14496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603317#comment-15603317 ] Jesus Camacho Rodriguez commented on HIVE-14496: [~ashutoshc], fails are unrelated. This is ready to be reviewed. Thanks > Enable Calcite rewriting with materialized views > > > Key: HIVE-14496 > URL: https://issues.apache.org/jira/browse/HIVE-14496 > Project: Hive > Issue Type: Sub-task > Components: Materialized views >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-14496.01.patch, HIVE-14496.02.patch, > HIVE-14496.patch > > > Calcite already supports query rewriting using materialized views. We will > use it to support this feature in Hive. > In order to do that, we need to register the existing materialized views with > Calcite view service and enable the materialized views rewriting rules. > We should include a HiveConf flag to completely disable query rewriting using > materialized views if necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15030) Fixes in inference of collation for Tez cost model
[ https://issues.apache.org/jira/browse/HIVE-15030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603315#comment-15603315 ] Jesus Camacho Rodriguez commented on HIVE-15030: [~ashutoshc], fails are unrelated. Could you review the patch? Thanks > Fixes in inference of collation for Tez cost model > -- > > Key: HIVE-15030 > URL: https://issues.apache.org/jira/browse/HIVE-15030 > Project: Hive > Issue Type: Bug > Components: Statistics >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-15030.patch > > > Tez cost model might get NPE if collation returned by join algorithm is null. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15046) Druid query type not recognized after Calcite upgrade
[ https://issues.apache.org/jira/browse/HIVE-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603255#comment-15603255 ] Jesus Camacho Rodriguez commented on HIVE-15046: [~ashutoshc], could you review this one? Since we do not have integration tests yet, I did not realize about it when we upgraded to Calcite 1.10. > Druid query type not recognized after Calcite upgrade > - > > Key: HIVE-15046 > URL: https://issues.apache.org/jira/browse/HIVE-15046 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-15046.patch > > > Introduced by HIVE-13316. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15046) Druid query type not recognized after Calcite upgrade
[ https://issues.apache.org/jira/browse/HIVE-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-15046: --- Attachment: HIVE-15046.patch > Druid query type not recognized after Calcite upgrade > - > > Key: HIVE-15046 > URL: https://issues.apache.org/jira/browse/HIVE-15046 > Project: Hive > Issue Type: Bug > Components: Druid integration >Affects Versions: 2.2.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-15046.patch > > > Introduced by HIVE-13316. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15025) Secure-Socket-Layer (SSL) support for HMS
[ https://issues.apache.org/jira/browse/HIVE-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603199#comment-15603199 ] Aihua Xu commented on HIVE-15025: - [~ctang.ma], [~xuefuz], [~spena], can you help review the patch? > Secure-Socket-Layer (SSL) support for HMS > - > > Key: HIVE-15025 > URL: https://issues.apache.org/jira/browse/HIVE-15025 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15025.1.patch, HIVE-15025.2.patch, > HIVE-15025.3.patch > > > HMS server should support SSL encryption. When the server is keberos enabled, > the encryption can be enabled. But if keberos is not enabled, then there is > no encryption between HS2 and HMS. > Similar to HS2, we should support encryption in both cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15025) Secure-Socket-Layer (SSL) support for HMS
[ https://issues.apache.org/jira/browse/HIVE-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15025: Attachment: HIVE-15025.3.patch Patch-3: add the unit test. > Secure-Socket-Layer (SSL) support for HMS > - > > Key: HIVE-15025 > URL: https://issues.apache.org/jira/browse/HIVE-15025 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 2.2.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-15025.1.patch, HIVE-15025.2.patch, > HIVE-15025.3.patch > > > HMS server should support SSL encryption. When the server is keberos enabled, > the encryption can be enabled. But if keberos is not enabled, then there is > no encryption between HS2 and HMS. > Similar to HS2, we should support encryption in both cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603171#comment-15603171 ] Sergio Peña commented on HIVE-15041: Sure. I'll do some testing on a ptest environment before commit it. > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603151#comment-15603151 ] Siddharth Seth commented on HIVE-15041: --- +1. [~spena] - could you please run the ptest unit tests before committing (they don't run on jenkins) > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15044) LockManager may be too coarse grained
[
https://issues.apache.org/jira/browse/HIVE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15044:
--
Description:
Consider
{noformat}
create table target (a int, b int)
partitioned by (p int, q int) clustered by (a) into 2 buckets
stored as orc TBLPROPERTIES ('transactional'='true')")
insert into target partition(p=1,q) values (1,2,3)
{noformat}
this insert will lock the whole table. See
{noformat}
DbTxnManager.acquireLocks()
switch (output.getType()) {
case DUMMYPARTITION: //
{noformat}
Insert operation runs with SHARED_READ lock but once HIVE-15032 is addressed
this will be an issue for Update/Delete/Merge which use a more restrictive
SHARED_WRITE lock.
This can probably be achieved using "like /db/table/part/*" predicate making
the LM operations more expensive TxnHandler.checkLock()
was:
Consider
{noformat}
create table target (a int, b int)
partitioned by (p int, q int) clustered by (a) into 2 buckets
stored as orc TBLPROPERTIES ('transactional'='true')")
insert into target partition(p=1,q) values (1,2,3)
{noformat}
this insert will lock the whole table. See
{noformat}
DbTxnManager.acquireLocks()
switch (output.getType()) {
case DUMMYPARTITION: //
{noformat}
Insert operation runs with SHARED_READ lock but once HIVE-15032 is addressed
this will be an issue for Update/Delete/Merge which use a more restrictive
SHARED_WRITE lock.
This can probably be achieved using "like /db/table/part/*" predicate making
the LM operations more expensive
> LockManager may be too coarse grained
> --
>
> Key: HIVE-15044
> URL: https://issues.apache.org/jira/browse/HIVE-15044
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> Consider
> {noformat}
> create table target (a int, b int)
> partitioned by (p int, q int) clustered by (a) into 2 buckets
> stored as orc TBLPROPERTIES ('transactional'='true')")
> insert into target partition(p=1,q) values (1,2,3)
> {noformat}
> this insert will lock the whole table. See
> {noformat}
> DbTxnManager.acquireLocks()
> switch (output.getType()) {
> case DUMMYPARTITION: //
> {noformat}
> Insert operation runs with SHARED_READ lock but once HIVE-15032 is addressed
> this will be an issue for Update/Delete/Merge which use a more restrictive
> SHARED_WRITE lock.
> This can probably be achieved using "like /db/table/part/*" predicate making
> the LM operations more expensive TxnHandler.checkLock()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15044) LockManager may be too coarse grained
[
https://issues.apache.org/jira/browse/HIVE-15044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-15044:
--
Description:
Consider
{noformat}
create table target (a int, b int)
partitioned by (p int, q int) clustered by (a) into 2 buckets
stored as orc TBLPROPERTIES ('transactional'='true')")
insert into target partition(p=1,q) values (1,2,3)
{noformat}
this insert will lock the whole table. See
{noformat}
DbTxnManager.acquireLocks()
switch (output.getType()) {
case DUMMYPARTITION: //
{noformat}
Insert operation runs with SHARED_READ lock but once HIVE-15032 is addressed
this will be an issue for Update/Delete/Merge which use a more restrictive
SHARED_WRITE lock.
This can probably be achieved using "like /db/table/part/*" predicate making
the LM operations more expensive
was:
Consider
{noformat}
create table target (a int, b int)
partitioned by (p int, q int) clustered by (a) into 2 buckets
stored as orc TBLPROPERTIES ('transactional'='true')")
insert into target partition(p=1,q) values (1,2,3)
{noformat}
this insert will lock the whole table. See
{noformat}
DbTxnManager.acquireLocks()
switch (output.getType()) {
case DUMMYPARTITION: //
{noformat}
Insert operation runs with SHARED_READ lock but once HIVE-15032 is addressed
this will be an issue for Update/Delete/Merge which use a more restrictive
SHARED_WRITE lock.
> LockManager may be too coarse grained
> --
>
> Key: HIVE-15044
> URL: https://issues.apache.org/jira/browse/HIVE-15044
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> Consider
> {noformat}
> create table target (a int, b int)
> partitioned by (p int, q int) clustered by (a) into 2 buckets
> stored as orc TBLPROPERTIES ('transactional'='true')")
> insert into target partition(p=1,q) values (1,2,3)
> {noformat}
> this insert will lock the whole table. See
> {noformat}
> DbTxnManager.acquireLocks()
> switch (output.getType()) {
> case DUMMYPARTITION: //
> {noformat}
> Insert operation runs with SHARED_READ lock but once HIVE-15032 is addressed
> this will be an issue for Update/Delete/Merge which use a more restrictive
> SHARED_WRITE lock.
> This can probably be achieved using "like /db/table/part/*" predicate making
> the LM operations more expensive
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10901) Optimize mutli column distinct queries
[ https://issues.apache.org/jira/browse/HIVE-10901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603097#comment-15603097 ] Ashutosh Chauhan commented on HIVE-10901: - Calcite has a rewrite for this query pattern using grouping-sets which was introduced in CALCITE-732 . However, while adapting that for hive ran into a limitation of Hive which is HIVE-15045 Currently, Hive doesn't support grouping sets when a column is part of both grouping set as well as aggregation function, which is what rewrite of CALCITE-732 will result in. This limitation is physical since it seems current GroupBy operator cannot handle this use case. > Optimize mutli column distinct queries > > > Key: HIVE-10901 > URL: https://issues.apache.org/jira/browse/HIVE-10901 > Project: Hive > Issue Type: New Feature > Components: CBO, Logical Optimizer >Affects Versions: 1.2.0 >Reporter: Mostafa Mokhtar >Assignee: Ashutosh Chauhan > Attachments: HIVE-10901.patch > > > HIVE-10568 is useful only when there is a distinct on one column. It can be > expanded for multiple column cases too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15045) Count distinct throws error when used in conjunction with grouping set
[
https://issues.apache.org/jira/browse/HIVE-15045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603076#comment-15603076
]
Ashutosh Chauhan commented on HIVE-15045:
-
Oracle which uses slightly different syntax allows both variants:
{code}
select empno, deptno, count(distinct empno) from scott.emp group by grouping
sets (empno, deptno);
select empno, deptno, count(distinct empno) from scott.emp group by cube
(empno, deptno);
{code}
> Count distinct throws error when used in conjunction with grouping set
> --
>
> Key: HIVE-15045
> URL: https://issues.apache.org/jira/browse/HIVE-15045
> Project: Hive
> Issue Type: Bug
> Components: Operators
>Affects Versions: 1.0.0, 1.2.0, 1.1.0, 2.0.0, 2.1.0
>Reporter: Ashutosh Chauhan
>
> {code}
> SELECT key, value, count(distinct value) FROM src GROUP BY key, value with
> cube;
> FAILED: SemanticException [Error 10210]: Grouping sets aggregations (with
> rollups or cubes) are not allowed if aggregation function parameters overlap
> with the aggregation functions columns
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13995) Hive metastore queries when there is no filter on partition column are inefficient
[
https://issues.apache.org/jira/browse/HIVE-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603066#comment-15603066
]
Ashutosh Chauhan commented on HIVE-13995:
-
[~thejas] Prior discussion and patches on this jira are not restricted for case
without filter.
It may be possible to optimize that specific case and may help benchmark, but
IMHO we should concentrate on general case first.
> Hive metastore queries when there is no filter on partition column are
> inefficient
> --
>
> Key: HIVE-13995
> URL: https://issues.apache.org/jira/browse/HIVE-13995
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13995.1.patch, HIVE-13995.2.patch,
> HIVE-13995.3.patch, HIVE-13995.4.patch, HIVE-13995.5.patch,
> HIVE-13995.6.patch, HIVE-13995.7.patch
>
>
> TPCDS fact tables (store_sales, catalog_sales) have 1800+ partitions and when
> the query does not a filter on the partition column, metastore queries
> generated have a large IN clause listing all the partition names. Most RDBMS
> systems have issues optimizing large IN clause and even when a good index
> plan is chosen , comparing to 1800+ string values will not lead to best
> execution time.
> When all partitions are chosen, not specifying the partition list and having
> filters only on table and column name will generate the same result set as
> long as there are no concurrent modifications to partition list of the hive
> table (adding/dropping partitions).
> For eg: For TPCDS query18, the metastore query gathering partition column
> statistics runs in 0.5 secs in Mysql. Following is output from mysql log
> {noformat}
> -- Query_time: 0.482063 Lock_time: 0.003037 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales'
> and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" in
> ('cs_sold_date_sk=2450815','cs_sold_date_sk=2450816','cs_sold_date_sk=2450817','cs_sold_date_sk=2450818','cs_sold_date_sk=2450819','cs_sold_date_sk=2450820','cs_sold_date_sk=2450821','cs_sold_date_sk=2450822','cs_sold_date_sk=2450823','cs_sold_date_sk=2450824','cs_sold_date_sk=2450825','cs_sold_date_sk=2450826','cs_sold_date_sk=2450827','cs_sold_date_sk=2450828','cs_sold_date_sk=2450829','cs_sold_date_sk=2450830','cs_sold_date_sk=2450831','cs_sold_date_sk=2450832','cs_sold_date_sk=2450833','cs_sold_date_sk=2450834','cs_sold_date_sk=2450835','cs_sold_date_sk=2450836','cs_sold_date_sk=2450837','cs_sold_date_sk=2450838','cs_sold_date_sk=2450839','cs_sold_date_sk=2450840','cs_sold_date_sk=2450841','cs_sold_date_sk=2450842','cs_sold_date_sk=2450843','cs_sold_date_sk=2450844','cs_sold_date_sk=2450845','cs_sold_date_sk=2450846','cs_sold_date_sk=2450847','cs_sold_date_sk=2450848','cs_sold_date_sk=2450849','cs_sold_date_sk=2450850','cs_sold_date_sk=2450851','cs_sold_date_sk=2450852','cs_sold_date_sk=2450853','cs_sold_date_sk=2450854','cs_sold_date_sk=2450855','cs_sold_date_sk=2450856',...,'cs_sold_date_sk=2452654')
> group by "PARTITION_NAME";
> {noformat}
> Functionally equivalent query runs in 0.1 seconds
> {noformat}
> --Query_time: 0.121296 Lock_time: 0.000156 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales' and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> group by "PARTITION_NAME";
> {noformat}
> If removing the partition list seems drastic, its also possible to simply
> list the range since hive gets a ordered list of partition names. This
> performs equally well as earlier query
> {noformat}
> # Query_time: 0.143874 Lock_time: 0.000154 Rows_sent: 1836 Rows_examined:
> 18360
> SET timestamp=1464014881;
> select count("COLUMN_NAME") from "PART_COL_STATS" where "DB_NAME" =
> 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" = 'catalog_sales' and
> "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" >= 'cs_sold_date_sk=2450815' and "PARTITION_NAME" <=
> 'cs_sold_date_sk=2452654'
> group by "PARTITION_NAME";
> {noformat}
> Another thing to check is the IN clause of column names. Columns in
> projection list of hive query are mentioned here. Not sure if statistics of
> these
[jira] [Updated] (HIVE-14909) Preserve the "parent location" of the table when an "alter table rename to " is submitted (the case when the db location is not specified and the Hive de
[ https://issues.apache.org/jira/browse/HIVE-14909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-14909: --- Attachment: HIVE-14909.1.patch Reattach patch to kick off the precommit tests > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). > -- > > Key: HIVE-14909 > URL: https://issues.apache.org/jira/browse/HIVE-14909 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 1.1.0 >Reporter: Adriano >Assignee: Chaoyu Tang > Attachments: HIVE-14909.1.patch, HIVE-14909.1.patch, > HIVE-14909.patch, HIVE-14909.patch > > > Alter Table operation for db_enc.rename_test failed to move data due to: > '/hdfs/encrypted_path/db_enc/rename_test can't be moved from an encryption > zone.' > When Hive renames a managed table, it always creates the new renamed table > directory under its database directory in order to keep a db/table hierarchy. > In this case, the renamed table directory is created under "default db" > directory "hive/warehouse/". When Hive renames a managed table, it always > creates the new renamed table directory under its database directory in order > to keep a db/table hierarchy. In this case, the renamed table directory is > created under "default' db directory typically set as /hive/warehouse/ . > This error doesn't appear if first create a database which points to a > directory outside /hive/warehouse/, say '/hdfs/encrypted_path', you won't > have this problem. For example, > create database db_enc location '/hdfs/encrypted_path/db_enc; > use db_enc; > create table rename_test (...) location > '/hdfs/encrypted_path/db_enc/rename_test'; > alter table rename_test rename to test_rename; > The renamed test_rename directory is created under > /hdfs/encrypted_path/db_enc. > Considering that the encryption of a filesystem is part of the evolution > hardening of a system (where the system and the data contained can already > exists) and a db can be already created without location set (because it is > not strictly required)and the default db is outside the same encryption zone > (or in a no-encryption zone) the alter table rename operation will fail. > Improvement: > Preserve the "parent location" of the table when an "alter table > rename to " is submitted (the case when the db location is not > specified and the Hive defult db is outside the same encrypted zone). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15033) Ensure there is only 1 StatsTask in the query plan
[ https://issues.apache.org/jira/browse/HIVE-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603049#comment-15603049 ] Eugene Koifman commented on HIVE-15033: --- this is only for the multi-insert which represents a MERGE statement, in which case multiple insert branches are writing to the same table/partition. There is no point computing stats until the whole statement is finished > Ensure there is only 1 StatsTask in the query plan > -- > > Key: HIVE-15033 > URL: https://issues.apache.org/jira/browse/HIVE-15033 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > currently there is 1 per WHEN clause -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15017) Random job failures with MapReduce and Tez
[ https://issues.apache.org/jira/browse/HIVE-15017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602905#comment-15602905 ] Sergey Shelukhin commented on HIVE-15017: - HADOOP logs. No, container-executor should be there, that is correct. This is what is (presumably) executed by the command that fails, assuming HADOOP_YARN_HOME is set and yarn.nodemanager.linux-container-executor.path doesn't override it - so, I wanted to double-check that the file is there. > Random job failures with MapReduce and Tez > -- > > Key: HIVE-15017 > URL: https://issues.apache.org/jira/browse/HIVE-15017 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 > Environment: Hadoop 2.7.2, Hive 2.1.0 >Reporter: Alexandre Linte >Priority: Critical > Attachments: hive-site.xml, hive_cli_mr.txt, hive_cli_tez.txt, > nodemanager_logs_mr_job.txt, yarn_container_tez_job_datanode05.txt, > yarn_container_tez_job_datanode06.txt, yarn_syslog_mr_job.txt, > yarn_syslog_tez_job.txt > > > Since Hive 2.1.0, we are facing a blocking issue on our cluster. All the jobs > are failing randomly on mapreduce and tez as well. > In both case, we don't have any ERROR or WARN message in the logs. You can > find attached: > - hive cli output errors > - yarn logs for a tez and mapreduce job > - nodemanager logs (mr only, we have the same logs with tez) > Note: This issue doesn't exist with Pig jobs (mr + tez), Spark jobs (mr), so > this cannot be an Hadoop / Yarn issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14645) table conversion to and from MM
[ https://issues.apache.org/jira/browse/HIVE-14645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-14645: --- Assignee: Sergey Shelukhin > table conversion to and from MM > --- > > Key: HIVE-14645 > URL: https://issues.apache.org/jira/browse/HIVE-14645 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15033) Ensure there is only 1 StatsTask in the query plan
[ https://issues.apache.org/jira/browse/HIVE-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602893#comment-15602893 ] Sergey Shelukhin commented on HIVE-15033: - Hmm... why cannot there be multiple stats tasks in case of multiple tables, multi-insert, etc.? > Ensure there is only 1 StatsTask in the query plan > -- > > Key: HIVE-15033 > URL: https://issues.apache.org/jira/browse/HIVE-15033 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > currently there is 1 per WHEN clause -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15038) Reuse OrcTail in OrcEncodedDataReader::ReaderOptions in Llap
[
https://issues.apache.org/jira/browse/HIVE-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602886#comment-15602886
]
Sergey Shelukhin commented on HIVE-15038:
-
+1 pending tests
> Reuse OrcTail in OrcEncodedDataReader::ReaderOptions in Llap
>
>
> Key: HIVE-15038
> URL: https://issues.apache.org/jira/browse/HIVE-15038
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15038.1.patch, HIVE-15038.2.patch
>
>
> When {{hive.orc.splits.include.file.footer}} is enabled, OrcTail (if
> available) can be made use of in OrcEncodedDataReader. This would reduce file
> look up calls (expensive in cloud storage systems like S3) and , eliminate
> footer reading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13995) Hive metastore queries when there is no filter on partition column are inefficient
[
https://issues.apache.org/jira/browse/HIVE-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602801#comment-15602801
]
Thejas M Nair commented on HIVE-13995:
--
Updating the jira summary to reflect the specific changes being discussed here.
> Hive metastore queries when there is no filter on partition column are
> inefficient
> --
>
> Key: HIVE-13995
> URL: https://issues.apache.org/jira/browse/HIVE-13995
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13995.1.patch, HIVE-13995.2.patch,
> HIVE-13995.3.patch, HIVE-13995.4.patch, HIVE-13995.5.patch,
> HIVE-13995.6.patch, HIVE-13995.7.patch
>
>
> TPCDS fact tables (store_sales, catalog_sales) have 1800+ partitions and when
> the query does not a filter on the partition column, metastore queries
> generated have a large IN clause listing all the partition names. Most RDBMS
> systems have issues optimizing large IN clause and even when a good index
> plan is chosen , comparing to 1800+ string values will not lead to best
> execution time.
> When all partitions are chosen, not specifying the partition list and having
> filters only on table and column name will generate the same result set as
> long as there are no concurrent modifications to partition list of the hive
> table (adding/dropping partitions).
> For eg: For TPCDS query18, the metastore query gathering partition column
> statistics runs in 0.5 secs in Mysql. Following is output from mysql log
> {noformat}
> -- Query_time: 0.482063 Lock_time: 0.003037 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales'
> and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" in
> ('cs_sold_date_sk=2450815','cs_sold_date_sk=2450816','cs_sold_date_sk=2450817','cs_sold_date_sk=2450818','cs_sold_date_sk=2450819','cs_sold_date_sk=2450820','cs_sold_date_sk=2450821','cs_sold_date_sk=2450822','cs_sold_date_sk=2450823','cs_sold_date_sk=2450824','cs_sold_date_sk=2450825','cs_sold_date_sk=2450826','cs_sold_date_sk=2450827','cs_sold_date_sk=2450828','cs_sold_date_sk=2450829','cs_sold_date_sk=2450830','cs_sold_date_sk=2450831','cs_sold_date_sk=2450832','cs_sold_date_sk=2450833','cs_sold_date_sk=2450834','cs_sold_date_sk=2450835','cs_sold_date_sk=2450836','cs_sold_date_sk=2450837','cs_sold_date_sk=2450838','cs_sold_date_sk=2450839','cs_sold_date_sk=2450840','cs_sold_date_sk=2450841','cs_sold_date_sk=2450842','cs_sold_date_sk=2450843','cs_sold_date_sk=2450844','cs_sold_date_sk=2450845','cs_sold_date_sk=2450846','cs_sold_date_sk=2450847','cs_sold_date_sk=2450848','cs_sold_date_sk=2450849','cs_sold_date_sk=2450850','cs_sold_date_sk=2450851','cs_sold_date_sk=2450852','cs_sold_date_sk=2450853','cs_sold_date_sk=2450854','cs_sold_date_sk=2450855','cs_sold_date_sk=2450856',...,'cs_sold_date_sk=2452654')
> group by "PARTITION_NAME";
> {noformat}
> Functionally equivalent query runs in 0.1 seconds
> {noformat}
> --Query_time: 0.121296 Lock_time: 0.000156 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales' and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> group by "PARTITION_NAME";
> {noformat}
> If removing the partition list seems drastic, its also possible to simply
> list the range since hive gets a ordered list of partition names. This
> performs equally well as earlier query
> {noformat}
> # Query_time: 0.143874 Lock_time: 0.000154 Rows_sent: 1836 Rows_examined:
> 18360
> SET timestamp=1464014881;
> select count("COLUMN_NAME") from "PART_COL_STATS" where "DB_NAME" =
> 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" = 'catalog_sales' and
> "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" >= 'cs_sold_date_sk=2450815' and "PARTITION_NAME" <=
> 'cs_sold_date_sk=2452654'
> group by "PARTITION_NAME";
> {noformat}
> Another thing to check is the IN clause of column names. Columns in
> projection list of hive query are mentioned here. Not sure if statistics of
> these columns are required for hive query optimization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13995) Hive metastore queries when there is no filter on partition column are inefficient
[
https://issues.apache.org/jira/browse/HIVE-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thejas M Nair updated HIVE-13995:
-
Summary: Hive metastore queries when there is no filter on partition column
are inefficient (was: Hive generates inefficient metastore queries for TPCDS
tables with 1800+ partitions leading to higher compile time)
> Hive metastore queries when there is no filter on partition column are
> inefficient
> --
>
> Key: HIVE-13995
> URL: https://issues.apache.org/jira/browse/HIVE-13995
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13995.1.patch, HIVE-13995.2.patch,
> HIVE-13995.3.patch, HIVE-13995.4.patch, HIVE-13995.5.patch,
> HIVE-13995.6.patch, HIVE-13995.7.patch
>
>
> TPCDS fact tables (store_sales, catalog_sales) have 1800+ partitions and when
> the query does not a filter on the partition column, metastore queries
> generated have a large IN clause listing all the partition names. Most RDBMS
> systems have issues optimizing large IN clause and even when a good index
> plan is chosen , comparing to 1800+ string values will not lead to best
> execution time.
> When all partitions are chosen, not specifying the partition list and having
> filters only on table and column name will generate the same result set as
> long as there are no concurrent modifications to partition list of the hive
> table (adding/dropping partitions).
> For eg: For TPCDS query18, the metastore query gathering partition column
> statistics runs in 0.5 secs in Mysql. Following is output from mysql log
> {noformat}
> -- Query_time: 0.482063 Lock_time: 0.003037 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales'
> and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" in
> ('cs_sold_date_sk=2450815','cs_sold_date_sk=2450816','cs_sold_date_sk=2450817','cs_sold_date_sk=2450818','cs_sold_date_sk=2450819','cs_sold_date_sk=2450820','cs_sold_date_sk=2450821','cs_sold_date_sk=2450822','cs_sold_date_sk=2450823','cs_sold_date_sk=2450824','cs_sold_date_sk=2450825','cs_sold_date_sk=2450826','cs_sold_date_sk=2450827','cs_sold_date_sk=2450828','cs_sold_date_sk=2450829','cs_sold_date_sk=2450830','cs_sold_date_sk=2450831','cs_sold_date_sk=2450832','cs_sold_date_sk=2450833','cs_sold_date_sk=2450834','cs_sold_date_sk=2450835','cs_sold_date_sk=2450836','cs_sold_date_sk=2450837','cs_sold_date_sk=2450838','cs_sold_date_sk=2450839','cs_sold_date_sk=2450840','cs_sold_date_sk=2450841','cs_sold_date_sk=2450842','cs_sold_date_sk=2450843','cs_sold_date_sk=2450844','cs_sold_date_sk=2450845','cs_sold_date_sk=2450846','cs_sold_date_sk=2450847','cs_sold_date_sk=2450848','cs_sold_date_sk=2450849','cs_sold_date_sk=2450850','cs_sold_date_sk=2450851','cs_sold_date_sk=2450852','cs_sold_date_sk=2450853','cs_sold_date_sk=2450854','cs_sold_date_sk=2450855','cs_sold_date_sk=2450856',...,'cs_sold_date_sk=2452654')
> group by "PARTITION_NAME";
> {noformat}
> Functionally equivalent query runs in 0.1 seconds
> {noformat}
> --Query_time: 0.121296 Lock_time: 0.000156 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales' and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> group by "PARTITION_NAME";
> {noformat}
> If removing the partition list seems drastic, its also possible to simply
> list the range since hive gets a ordered list of partition names. This
> performs equally well as earlier query
> {noformat}
> # Query_time: 0.143874 Lock_time: 0.000154 Rows_sent: 1836 Rows_examined:
> 18360
> SET timestamp=1464014881;
> select count("COLUMN_NAME") from "PART_COL_STATS" where "DB_NAME" =
> 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" = 'catalog_sales' and
> "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" >= 'cs_sold_date_sk=2450815' and "PARTITION_NAME" <=
> 'cs_sold_date_sk=2452654'
> group by "PARTITION_NAME";
> {noformat}
> Another thing to check is the IN clause of column names. Columns in
> projection list of hive query are mentioned here. Not sure if statistics of
> these columns are required for hive query optimization.
[jira] [Updated] (HIVE-13589) beeline support prompt for password with '-p' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-13589: --- Attachment: HIVE-13589.12.patch For some reason HiveQA job is not picking up the patch since last two days. Will re-attach it to see if it helps. > beeline support prompt for password with '-p' option > > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Vihang Karajgaonkar > Fix For: 2.2.0 > > Attachments: HIVE-13589.1.patch, HIVE-13589.10.patch, > HIVE-13589.11.patch, HIVE-13589.12.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, > HIVE-13589.6.patch, HIVE-13589.7.patch, HIVE-13589.8.patch, HIVE-13589.9.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15042) Support intersect/except without distinct keyword
[ https://issues.apache.org/jira/browse/HIVE-15042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602704#comment-15602704 ] Ashutosh Chauhan commented on HIVE-15042: - +1 > Support intersect/except without distinct keyword > - > > Key: HIVE-15042 > URL: https://issues.apache.org/jira/browse/HIVE-15042 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15042.01.patch > > > basically, intersect = intersect distinct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9941) sql std authorization on partitioned table: truncate and insert
[
https://issues.apache.org/jira/browse/HIVE-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-9941:
-
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Committed to master.
> sql std authorization on partitioned table: truncate and insert
> ---
>
> Key: HIVE-9941
> URL: https://issues.apache.org/jira/browse/HIVE-9941
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.0.0, 1.2.0
>Reporter: Olaf Flebbe
>Assignee: Sushanth Sowmyan
> Fix For: 2.2.0
>
> Attachments: HIVE-9941.2.patch, HIVE-9941.3.patch, HIVE-9941.patch
>
>
> sql std authorization works as expected.
> However if a table is partitioned any user can truncate it
> User foo:
> {code}
> create table bla (a string) partitioned by (b string);
> #.. loading values ...
> {code}
> Admin:
> {code}
> 0: jdbc:hive2://localhost:1/default> set role admin;
> No rows affected (0,074 seconds)
> 0: jdbc:hive2://localhost:1/default> show grant on bla;
> +---+++-+-+-++---++--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---++--+--+
> | default | bla|| | foo | USER
> | DELETE | true | 1426158997000 | foo |
> | default | bla|| | foo | USER
> | INSERT | true | 1426158997000 | foo |
> | default | bla|| | foo | USER
> | SELECT | true | 1426158997000 | foo |
> | default | bla|| | foo | USER
> | UPDATE | true | 1426158997000 | foo |
> +---+++-+-+-++---++--+--+
> {code}
> now user olaf
> {code}
> 0: jdbc:hive2://localhost:1/default> select * from bla;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: Principal [name=olaf, type=USER] does not have following
> privileges for operation QUERY [[SELECT] on Object [type=TABLE_OR_VIEW,
> name=default.bla]] (state=42000,code=4)
> {code}
> works as expected.
> _BUT_
> {code}
> 0: jdbc:hive2://localhost:1/default> truncate table bla;
> No rows affected (0,18 seconds)
> {code}
> _And table is empty afterwards_.
> Similarily: {{insert into table}} works, too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15042) Support intersect/except without distinct keyword
[ https://issues.apache.org/jira/browse/HIVE-15042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15042: --- Attachment: HIVE-15042.01.patch > Support intersect/except without distinct keyword > - > > Key: HIVE-15042 > URL: https://issues.apache.org/jira/browse/HIVE-15042 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15042.01.patch > > > basically, intersect = intersect distinct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15042) Support intersect/except without distinct keyword
[ https://issues.apache.org/jira/browse/HIVE-15042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15042: --- Status: Patch Available (was: Open) as it is a simple patch, [~ashutoshc] could u take a look? Thanks. > Support intersect/except without distinct keyword > - > > Key: HIVE-15042 > URL: https://issues.apache.org/jira/browse/HIVE-15042 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-15042.01.patch > > > basically, intersect = intersect distinct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15034) Fix orc_ppd_basic & current_date_timestamp tests
[ https://issues.apache.org/jira/browse/HIVE-15034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-15034: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Vineet! > Fix orc_ppd_basic & current_date_timestamp tests > > > Key: HIVE-15034 > URL: https://issues.apache.org/jira/browse/HIVE-15034 > Project: Hive > Issue Type: Test > Components: Test >Reporter: Vineet Garg >Assignee: Vineet Garg > Fix For: 2.2.0 > > Attachments: HIVE-15034.1.patch > > > Started failing following HIVE-14913's failure -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15042) Support intersect/except without distinct keyword
[ https://issues.apache.org/jira/browse/HIVE-15042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-15042: --- Description: basically, intersect = intersect distinct. > Support intersect/except without distinct keyword > - > > Key: HIVE-15042 > URL: https://issues.apache.org/jira/browse/HIVE-15042 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > > basically, intersect = intersect distinct. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15003) Update 'ALTER TABLE...UPDATE STATISTICS FOR COLUMN..' statement to support more data types
[ https://issues.apache.org/jira/browse/HIVE-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602567#comment-15602567 ] Ashutosh Chauhan commented on HIVE-15003: - +1 > Update 'ALTER TABLE...UPDATE STATISTICS FOR COLUMN..' statement to support > more data types > -- > > Key: HIVE-15003 > URL: https://issues.apache.org/jira/browse/HIVE-15003 > Project: Hive > Issue Type: Task > Components: Statistics >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-15003.1.patch > > > Currently ALTER TABLE...UDPATE STATISTICS FOR COLUMN... only support updating > statistics for following data types: > * STRING > * DOUBLE > * BOOLEAN > * BINARY > * DECIMAL > * DATE > We are missing the following data types > * TINYINT > * SMALLINT > * INT > * BIGINT > * FLOAT > * VARCHAR > * CHAR > * TIMESTAMP -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11208) Can not drop a default partition __HIVE_DEFAULT_PARTITION__ which is not a "string" type
[
https://issues.apache.org/jira/browse/HIVE-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15602513#comment-15602513
]
Hive QA commented on HIVE-11208:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12834936/HIVE-11208.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10583 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=131)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[current_date_timestamp]
(batchId=144)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0]
(batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=164)
org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=164)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1768/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1768/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-1768/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12834936 - PreCommit-HIVE-Build
> Can not drop a default partition __HIVE_DEFAULT_PARTITION__ which is not a
> "string" type
>
>
> Key: HIVE-11208
> URL: https://issues.apache.org/jira/browse/HIVE-11208
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Aihua Xu
> Attachments: HIVE-11208.2.patch
>
>
> When partition is not a string type, for example, if it is a int type, when
> drop the default partition __HIVE_DEFAULT_PARTITION__, you will get:
> SemanticException Unexpected unknown partitions
> Reproduce:
> {noformat}
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
> set hive.exec.max.dynamic.partitions.pernode=1;
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (col1 string) PARTITIONED BY (p1 int) ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\001' STORED AS TEXTFILE;
> INSERT OVERWRITE TABLE test PARTITION (p1) SELECT code, IF(salary > 600, 100,
> null) as p1 FROM jsmall;
> hive> SHOW PARTITIONS test;
> OK
> p1=100
> p1=__HIVE_DEFAULT_PARTITION__
> Time taken: 0.124 seconds, Fetched: 2 row(s)
> hive> ALTER TABLE test DROP partition (p1 = '__HIVE_DEFAULT_PARTITION__');
> FAILED: SemanticException Unexpected unknown partitions for (p1 = null)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-15041: --- Attachment: HIVE-15041.1.patch [~sseth] [~prasanth_j] Could you review please? > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-15041: --- Status: Patch Available (was: Open) > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-15041.1.patch > > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15041) Specify GCE network name on Hive ptest
[ https://issues.apache.org/jira/browse/HIVE-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-15041: --- Description: NO PRECOMMIT TESTS A new option on cloudhost.properties should be added to specify the GCE network name: # GCE network option network = https://www.googleapis.com/compute/v1/projects//global/networks/default was: A new option on cloudhost.properties should be added to specify the GCE network name: # GCE network option network = https://www.googleapis.com/compute/v1/projects//global/networks/default > Specify GCE network name on Hive ptest > -- > > Key: HIVE-15041 > URL: https://issues.apache.org/jira/browse/HIVE-15041 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña > > NO PRECOMMIT TESTS > A new option on cloudhost.properties should be added to specify the GCE > network name: > # GCE network option > network = > https://www.googleapis.com/compute/v1/projects//global/networks/default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14753) Track the number of open/closed/abandoned sessions in HS2
[ https://issues.apache.org/jira/browse/HIVE-14753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-14753: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master, thanks Barna for the contribution! > Track the number of open/closed/abandoned sessions in HS2 > - > > Key: HIVE-14753 > URL: https://issues.apache.org/jira/browse/HIVE-14753 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2 >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Fix For: 2.2.0 > > Attachments: HIVE-14753.1.patch, HIVE-14753.2.patch, > HIVE-14753.3.patch, HIVE-14753.patch > > > We should be able to track the nr. of sessions since the startup of the HS2 > instance as well as the average lifetime of a session. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11208) Can not drop a default partition __HIVE_DEFAULT_PARTITION__ which is not a "string" type
[
https://issues.apache.org/jira/browse/HIVE-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-11208:
Status: Patch Available (was: Open)
> Can not drop a default partition __HIVE_DEFAULT_PARTITION__ which is not a
> "string" type
>
>
> Key: HIVE-11208
> URL: https://issues.apache.org/jira/browse/HIVE-11208
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Aihua Xu
> Attachments: HIVE-11208.2.patch
>
>
> When partition is not a string type, for example, if it is a int type, when
> drop the default partition __HIVE_DEFAULT_PARTITION__, you will get:
> SemanticException Unexpected unknown partitions
> Reproduce:
> {noformat}
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
> set hive.exec.max.dynamic.partitions.pernode=1;
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (col1 string) PARTITIONED BY (p1 int) ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\001' STORED AS TEXTFILE;
> INSERT OVERWRITE TABLE test PARTITION (p1) SELECT code, IF(salary > 600, 100,
> null) as p1 FROM jsmall;
> hive> SHOW PARTITIONS test;
> OK
> p1=100
> p1=__HIVE_DEFAULT_PARTITION__
> Time taken: 0.124 seconds, Fetched: 2 row(s)
> hive> ALTER TABLE test DROP partition (p1 = '__HIVE_DEFAULT_PARTITION__');
> FAILED: SemanticException Unexpected unknown partitions for (p1 = null)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11208) Can not drop a default partition __HIVE_DEFAULT_PARTITION__ which is not a "string" type
[
https://issues.apache.org/jira/browse/HIVE-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-11208:
Attachment: (was: HIVE-11208.1.patch)
> Can not drop a default partition __HIVE_DEFAULT_PARTITION__ which is not a
> "string" type
>
>
> Key: HIVE-11208
> URL: https://issues.apache.org/jira/browse/HIVE-11208
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Aihua Xu
> Attachments: HIVE-11208.2.patch
>
>
> When partition is not a string type, for example, if it is a int type, when
> drop the default partition __HIVE_DEFAULT_PARTITION__, you will get:
> SemanticException Unexpected unknown partitions
> Reproduce:
> {noformat}
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
> set hive.exec.max.dynamic.partitions.pernode=1;
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (col1 string) PARTITIONED BY (p1 int) ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\001' STORED AS TEXTFILE;
> INSERT OVERWRITE TABLE test PARTITION (p1) SELECT code, IF(salary > 600, 100,
> null) as p1 FROM jsmall;
> hive> SHOW PARTITIONS test;
> OK
> p1=100
> p1=__HIVE_DEFAULT_PARTITION__
> Time taken: 0.124 seconds, Fetched: 2 row(s)
> hive> ALTER TABLE test DROP partition (p1 = '__HIVE_DEFAULT_PARTITION__');
> FAILED: SemanticException Unexpected unknown partitions for (p1 = null)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11208) Can not drop a default partition __HIVE_DEFAULT_PARTITION__ which is not a "string" type
[
https://issues.apache.org/jira/browse/HIVE-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-11208:
Attachment: HIVE-11208.2.patch
> Can not drop a default partition __HIVE_DEFAULT_PARTITION__ which is not a
> "string" type
>
>
> Key: HIVE-11208
> URL: https://issues.apache.org/jira/browse/HIVE-11208
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Aihua Xu
> Attachments: HIVE-11208.2.patch
>
>
> When partition is not a string type, for example, if it is a int type, when
> drop the default partition __HIVE_DEFAULT_PARTITION__, you will get:
> SemanticException Unexpected unknown partitions
> Reproduce:
> {noformat}
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
> set hive.exec.max.dynamic.partitions.pernode=1;
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (col1 string) PARTITIONED BY (p1 int) ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\001' STORED AS TEXTFILE;
> INSERT OVERWRITE TABLE test PARTITION (p1) SELECT code, IF(salary > 600, 100,
> null) as p1 FROM jsmall;
> hive> SHOW PARTITIONS test;
> OK
> p1=100
> p1=__HIVE_DEFAULT_PARTITION__
> Time taken: 0.124 seconds, Fetched: 2 row(s)
> hive> ALTER TABLE test DROP partition (p1 = '__HIVE_DEFAULT_PARTITION__');
> FAILED: SemanticException Unexpected unknown partitions for (p1 = null)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-11208) Can not drop a default partition __HIVE_DEFAULT_PARTITION__ which is not a "string" type
[
https://issues.apache.org/jira/browse/HIVE-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-11208:
Status: Open (was: Patch Available)
> Can not drop a default partition __HIVE_DEFAULT_PARTITION__ which is not a
> "string" type
>
>
> Key: HIVE-11208
> URL: https://issues.apache.org/jira/browse/HIVE-11208
> Project: Hive
> Issue Type: Bug
> Components: Parser
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Aihua Xu
> Attachments: HIVE-11208.1.patch
>
>
> When partition is not a string type, for example, if it is a int type, when
> drop the default partition __HIVE_DEFAULT_PARTITION__, you will get:
> SemanticException Unexpected unknown partitions
> Reproduce:
> {noformat}
> SET hive.exec.dynamic.partition=true;
> SET hive.exec.dynamic.partition.mode=nonstrict;
> set hive.exec.max.dynamic.partitions.pernode=1;
> DROP TABLE IF EXISTS test;
> CREATE TABLE test (col1 string) PARTITIONED BY (p1 int) ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\001' STORED AS TEXTFILE;
> INSERT OVERWRITE TABLE test PARTITION (p1) SELECT code, IF(salary > 600, 100,
> null) as p1 FROM jsmall;
> hive> SHOW PARTITIONS test;
> OK
> p1=100
> p1=__HIVE_DEFAULT_PARTITION__
> Time taken: 0.124 seconds, Fetched: 2 row(s)
> hive> ALTER TABLE test DROP partition (p1 = '__HIVE_DEFAULT_PARTITION__');
> FAILED: SemanticException Unexpected unknown partitions for (p1 = null)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14580) Introduce || operator
[ https://issues.apache.org/jira/browse/HIVE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15601822#comment-15601822 ] Zoltan Haindrich commented on HIVE-14580: - [~leftylev], [~pxiong] - AFAIK there is no || or && logical operators...so i've removed them from the docs...i've peeked into branch-1; but i think there isn't any on that branch either...so I guess this was planned at some time; but never implemented. I think i've added the "needed" documentation for this operator. I've also added some first version of the precedence table: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-Operatorsprecedences > Introduce || operator > - > > Key: HIVE-14580 > URL: https://issues.apache.org/jira/browse/HIVE-14580 > Project: Hive > Issue Type: Sub-task > Components: SQL >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Zoltan Haindrich > Labels: TODOC2.2 > Fix For: 2.2.0 > > Attachments: HIVE-14580.1.patch, HIVE-14580.2.patch, > HIVE-14580.3.patch, HIVE-14580.4.patch > > > Functionally equivalent to concat() udf. But standard allows usage of || for > string concatenations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13830) Hive on spark driver crash with Spark 1.6.1
[
https://issues.apache.org/jira/browse/HIVE-13830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15601515#comment-15601515
]
KaiXu commented on HIVE-13830:
--
I used spark1.6.2 release version, spark1.6.4 and Hive 1.2.1, it has the same
error.
> Hive on spark driver crash with Spark 1.6.1
> ---
>
> Key: HIVE-13830
> URL: https://issues.apache.org/jira/browse/HIVE-13830
> Project: Hive
> Issue Type: Bug
> Components: Spark, spark-branch
>Affects Versions: 2.0.0, 2.1.0
> Environment: Hadoop 2.7.2, Hive 2.1.0, Spark 1.6.1, Kerberos
>Reporter: Alexandre Linte
>
> With Hive 1.2.1 I was able to use Hive on successfully with the use of the
> -assembly "-assembly-1.4.1-hadoop2.7.1.jar".
> Today with Hive 2.0.0, I'm unable to use Hive on whether it be with the
> -assembly "-assembly-1.4.1-hadoop2.7.1.jar" or the -assembly
> "-assembly-1.6.1-hadoop2.7.2.jar".
> My configuration is the following:
> * -. available in HIVE_DIR/
> * assembly available in HIVE_DIR/lib
> I gathered several logs below:
> - HQL commands
> {noformat}
> $ hive -v --database shfs3453
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hive/apache-hive-2.0.0-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application//-1.6.1/assembly/target/scala-2.10/-assembly-1.6.1-hadoop2.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/application/Hadoop/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Logging initialized using configuration in
> file:/opt/application/Hive/apache-hive-2.0.0-bin/conf/hive-log4j2.properties
> use shfs3453
> OK
> Time taken: 1.425 seconds
> Hive-on-MR is deprecated in Hive 2 and may not be available in the future
> versions. Consider using a different execution engine (i.e. tez, ) or using
> Hive 1.X releases.
> hive (shfs3453)> set hive.execution.engine=;
> set hive.execution.engine=
> hive (shfs3453)> set .master=yarn-client;
> set .master=yarn-client
> hive (shfs3453)> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING,
> Day STRING, Block STRING, IUCR INT, PrimaryType STRING, Description STRING,
> LocationDescription STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT,
> District INT, Ward INT, CommunityArea INT, FBICode INT, XCoordinate BIGINT,
> YCoordinate BIGINT, Year INT, UpdatedOn STRING, Latitude FLOAT, Longitude
> FLOAT, Location STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED
> AS TEXTFILE;
> CREATE TABLE chicagoCrimes2 (ID BIGINT, CaseNumber STRING, Day STRING, Block
> STRING, IUCR INT, PrimaryType STRING, Description STRING, LocationDescription
> STRING, Arrest BOOLEAN, Domestic BOOLEAN, Beat INT, District INT, Ward INT,
> CommunityArea INT, FBICode INT, XCoordinate BIGINT, YCoordinate BIGINT, Year
> INT, UpdatedOn STRING, Latitude FLOAT, Longitude FLOAT, Location STRING) ROW
> FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE
> OK
> Time taken: 0.408 seconds
> hive (shfs3453)> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM
> chicagocrimes WHERE Description = 'FIRST DEGREE MURDER';
> INSERT OVERWRITE TABLE chicagocrimes2 SELECT * FROM chicagocrimes WHERE
> Description = 'FIRST DEGREE MURDER'
> Query ID = shfs3453_20160524092714_41c89aec-2c6f-49e9-98c7-d227ca144f73
> Total jobs = 1
> Launching Job 1 out of 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=
> Starting Job = 79484279-8e75-4b13-8e71-7de463f4d51e
> Status: SENT
> Failed to execute task, with exception 'java.lang.IllegalStateException(RPC
> channel is closed.)'
> FAILED: Execution , return code 1 from
> org.apache.hadoop.hive.ql.exec..SparkTask
> {noformat}
> - Client logs
> {noformat}
> May 24 09:32:19 hive-cli - org.apache.hive..client.rpc.RpcDispatcherReceived
> message:io.netty.handler.codec.DecoderException:
> java.lang.NoClassDefFoundError: org/apache/hive//client/Job
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:358)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:230)
> at
>
[jira] [Updated] (HIVE-15038) Reuse OrcTail in OrcEncodedDataReader::ReaderOptions in Llap
[
https://issues.apache.org/jira/browse/HIVE-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rajesh Balamohan updated HIVE-15038:
Attachment: HIVE-15038.2.patch
Thanks [~gopalv]. Attaching revised patch.
> Reuse OrcTail in OrcEncodedDataReader::ReaderOptions in Llap
>
>
> Key: HIVE-15038
> URL: https://issues.apache.org/jira/browse/HIVE-15038
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-15038.1.patch, HIVE-15038.2.patch
>
>
> When {{hive.orc.splits.include.file.footer}} is enabled, OrcTail (if
> available) can be made use of in OrcEncodedDataReader. This would reduce file
> look up calls (expensive in cloud storage systems like S3) and , eliminate
> footer reading.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-14391) TestAccumuloCliDriver is not executed during precommit tests
[
https://issues.apache.org/jira/browse/HIVE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15601352#comment-15601352
]
Peter Vary edited comment on HIVE-14391 at 10/24/16 8:34 AM:
-
Hi [~ashutoshc],
After your proposal I have contacted the our product managers, and they
redirected me the following URL:
http://community.cloudera.com/t5/Community-News-Release/ANNOUNCE-Apache-Accumulo-1-7-Support-for-CDH-5/m-p/46142#U46142
Is short:
{code}
"[ANNOUNCE] Apache Accumulo 1.7 Support for CDH 5 (10-11-2016 07:14 AM)
[..]
This release adds support for:
[..]
- AccumuloStorgeHandler for Apache Hive access to data stored in Accumulo"
{code}
So by chance I have enabled tests which are quiet important for our company at
the moment.
If it is ok with the other apache project members, I think we would like to
keep this feature and enable at least these token tests for it in the apache
project as well.
Thanks,
Peter
PS: Is it ok to post this specific URLs on an apache JIRA? Or should I remove
it?
edit: formatting
was (Author: pvary):
Hi [~ashutoshc],
After your proposal I have contacted the our product managers, and they
redirected me the following URL:
http://community.cloudera.com/t5/Community-News-Release/ANNOUNCE-Apache-Accumulo-1-7-Support-for-CDH-5/m-p/46142#U46142
Is short:
"[ANNOUNCE] Apache Accumulo 1.7 Support for CDH 5 (10-11-2016 07:14 AM)
[..]
This release adds support for:
[..]
- AccumuloStorgeHandler for Apache Hive access to data stored in Accumulo"
So by chance I have enabled tests which are quiet important for our company at
the moment.
If it is ok with the other apache project members, I think we would like to
keep this feature and enable at least these token tests for it in the apache
project as well.
Thanks,
Peter
PS: Is it ok to post this specific URLs on an apache JIRA? Or should I remove
it?
> TestAccumuloCliDriver is not executed during precommit tests
>
>
> Key: HIVE-14391
> URL: https://issues.apache.org/jira/browse/HIVE-14391
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Peter Vary
> Attachments: HIVE-14391.2.patch, HIVE-14391.patch
>
>
> according to for example this build result:
> https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/685/testReport/org.apache.hadoop.hive.cli/
> there is no 'TestAccumuloCliDriver' being run during precommit testing...but
> i see no reason why and how it was excluded inside the project;
> my maven executes it when i start it with {{-Dtest=TestAccumuloCliDriver}} -
> so i think the properties/profiles aren't preventing it.
> maybe i miss something obvious ;)
> (note: my TestAccumuloCliDriver executions are failed with errors.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14391) TestAccumuloCliDriver is not executed during precommit tests
[
https://issues.apache.org/jira/browse/HIVE-14391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15601352#comment-15601352
]
Peter Vary commented on HIVE-14391:
---
Hi [~ashutoshc],
After your proposal I have contacted the our product managers, and they
redirected me the following URL:
http://community.cloudera.com/t5/Community-News-Release/ANNOUNCE-Apache-Accumulo-1-7-Support-for-CDH-5/m-p/46142#U46142
Is short:
"[ANNOUNCE] Apache Accumulo 1.7 Support for CDH 5 (10-11-2016 07:14 AM)
[..]
This release adds support for:
[..]
- AccumuloStorgeHandler for Apache Hive access to data stored in Accumulo"
So by chance I have enabled tests which are quiet important for our company at
the moment.
If it is ok with the other apache project members, I think we would like to
keep this feature and enable at least these token tests for it in the apache
project as well.
Thanks,
Peter
PS: Is it ok to post this specific URLs on an apache JIRA? Or should I remove
it?
> TestAccumuloCliDriver is not executed during precommit tests
>
>
> Key: HIVE-14391
> URL: https://issues.apache.org/jira/browse/HIVE-14391
> Project: Hive
> Issue Type: Sub-task
> Components: Testing Infrastructure
>Reporter: Zoltan Haindrich
>Assignee: Peter Vary
> Attachments: HIVE-14391.2.patch, HIVE-14391.patch
>
>
> according to for example this build result:
> https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/685/testReport/org.apache.hadoop.hive.cli/
> there is no 'TestAccumuloCliDriver' being run during precommit testing...but
> i see no reason why and how it was excluded inside the project;
> my maven executes it when i start it with {{-Dtest=TestAccumuloCliDriver}} -
> so i think the properties/profiles aren't preventing it.
> maybe i miss something obvious ;)
> (note: my TestAccumuloCliDriver executions are failed with errors.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15017) Random job failures with MapReduce and Tez
[ https://issues.apache.org/jira/browse/HIVE-15017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexandre Linte updated HIVE-15017: --- Attachment: hive-site.xml > Random job failures with MapReduce and Tez > -- > > Key: HIVE-15017 > URL: https://issues.apache.org/jira/browse/HIVE-15017 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 > Environment: Hadoop 2.7.2, Hive 2.1.0 >Reporter: Alexandre Linte >Priority: Critical > Attachments: hive-site.xml, hive_cli_mr.txt, hive_cli_tez.txt, > nodemanager_logs_mr_job.txt, yarn_container_tez_job_datanode05.txt, > yarn_container_tez_job_datanode06.txt, yarn_syslog_mr_job.txt, > yarn_syslog_tez_job.txt > > > Since Hive 2.1.0, we are facing a blocking issue on our cluster. All the jobs > are failing randomly on mapreduce and tez as well. > In both case, we don't have any ERROR or WARN message in the logs. You can > find attached: > - hive cli output errors > - yarn logs for a tez and mapreduce job > - nodemanager logs (mr only, we have the same logs with tez) > Note: This issue doesn't exist with Pig jobs (mr + tez), Spark jobs (mr), so > this cannot be an Hadoop / Yarn issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15017) Random job failures with MapReduce and Tez
[ https://issues.apache.org/jira/browse/HIVE-15017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15601270#comment-15601270 ] Alexandre Linte commented on HIVE-15017: Which logs do you need in DEBUG mode? (Hadoop, Hive) I added the hive-site.xml to help. The HADOOP_YARN_HOME env variable is properly set on every datanode. I checked. There is no "yarn.nodemanager.linux-container-executor.path" property set on any of Hadoop configuration files. The default value must be used. Yes, bin/container-executor is under Yarn home. Is that wrong? > Random job failures with MapReduce and Tez > -- > > Key: HIVE-15017 > URL: https://issues.apache.org/jira/browse/HIVE-15017 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 > Environment: Hadoop 2.7.2, Hive 2.1.0 >Reporter: Alexandre Linte >Priority: Critical > Attachments: hive_cli_mr.txt, hive_cli_tez.txt, > nodemanager_logs_mr_job.txt, yarn_container_tez_job_datanode05.txt, > yarn_container_tez_job_datanode06.txt, yarn_syslog_mr_job.txt, > yarn_syslog_tez_job.txt > > > Since Hive 2.1.0, we are facing a blocking issue on our cluster. All the jobs > are failing randomly on mapreduce and tez as well. > In both case, we don't have any ERROR or WARN message in the logs. You can > find attached: > - hive cli output errors > - yarn logs for a tez and mapreduce job > - nodemanager logs (mr only, we have the same logs with tez) > Note: This issue doesn't exist with Pig jobs (mr + tez), Spark jobs (mr), so > this cannot be an Hadoop / Yarn issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
