[jira] [Updated] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9283:
Fix Version/s: encryption-branch
Status: Patch Available (was: Open)
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9283.patch
>
>
> NO PRECOMMIT TESTS
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key. (Put in HIVE-9286)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9283:
Description:
NO PRECOMMIT TESTS
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key. (Put in HIVE-9286)
was:
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key. (Put in HIVE-9286)
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9283.patch
>
>
> NO PRECOMMIT TESTS
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key. (Put in HIVE-9286)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9283:
Attachment: HIVE-9283.patch
This patch update the table location in the .q files
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9283.patch
>
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key. (Put in HIVE-9286)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9261) Update the output files for the encryption qtests since the output format changed
[ https://issues.apache.org/jira/browse/HIVE-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9261: --- Assignee: Dong Chen (was: Ferdinand Xu) > Update the output files for the encryption qtests since the output format > changed > - > > Key: HIVE-9261 > URL: https://issues.apache.org/jira/browse/HIVE-9261 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Dong Chen > > Should update the output files due to the output format changes in HIVE-9254 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9287) Only 3 encryption test cases was run. The test configuration is not correct
[ https://issues.apache.org/jira/browse/HIVE-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9287: Attachment: HIVE-9287.patch A small change attached. The comma is missed, so that some cases is not parsed and added expected. > Only 3 encryption test cases was run. The test configuration is not correct > --- > > Key: HIVE-9287 > URL: https://issues.apache.org/jira/browse/HIVE-9287 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9287.patch > > > NO PRECOMMIT TESTS > Only 3 encryption test cases was run. The test configuration is not correct. > We should add all test case file in testconfiguration.properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9285) The move task doesn't work for inserting overwrite a local directory in test mode
[ https://issues.apache.org/jira/browse/HIVE-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9285: --- Description: In the query test, the data warehouse is using the file system instead of the hdfs. So the schema will be "file://" instead of "hdfs://" that results isSub method doesn't function well. We can reproduce this bug by running the command " FROM src INSERT OVERWRITE DIRECTORY 'target/warehouse/dest4.out' SELECT src.value WHERE src.key >= 300; " was:In the query test, the data warehouse is using the file system instead of the hdfs. So the schema will be "file://" instead of "hdfs://" that results isSub method doesn't function well. > The move task doesn't work for inserting overwrite a local directory in test > mode > - > > Key: HIVE-9285 > URL: https://issues.apache.org/jira/browse/HIVE-9285 > Project: Hive > Issue Type: Sub-task >Affects Versions: encryption-branch >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: encryption-branch > > Attachments: HIVE-9285-encryption.patch > > > In the query test, the data warehouse is using the file system instead of the > hdfs. So the schema will be "file://" instead of "hdfs://" that results isSub > method doesn't function well. > We can reproduce this bug by running the command > " > FROM src > INSERT OVERWRITE DIRECTORY 'target/warehouse/dest4.out' SELECT src.value > WHERE src.key >= 300; > " -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9287) Only 3 encryption test cases was run. The test configuration is not correct
[ https://issues.apache.org/jira/browse/HIVE-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9287: Description: NO PRECOMMIT TESTS Only 3 encryption test cases was run. The test configuration is not correct. We should add all test case file in testconfiguration.properties was: Only 3 encryption test cases was run. The test configuration is not correct. We should add all test case file in testconfiguration.properties > Only 3 encryption test cases was run. The test configuration is not correct > --- > > Key: HIVE-9287 > URL: https://issues.apache.org/jira/browse/HIVE-9287 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > > NO PRECOMMIT TESTS > Only 3 encryption test cases was run. The test configuration is not correct. > We should add all test case file in testconfiguration.properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9287) Only 3 encryption test cases was run. The test configuration is not correct
[ https://issues.apache.org/jira/browse/HIVE-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9287: Status: Patch Available (was: Open) > Only 3 encryption test cases was run. The test configuration is not correct > --- > > Key: HIVE-9287 > URL: https://issues.apache.org/jira/browse/HIVE-9287 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > > Only 3 encryption test cases was run. The test configuration is not correct. > We should add all test case file in testconfiguration.properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9287) Only 3 encryption test cases was run. The test configuration is not correct
Dong Chen created HIVE-9287: --- Summary: Only 3 encryption test cases was run. The test configuration is not correct Key: HIVE-9287 URL: https://issues.apache.org/jira/browse/HIVE-9287 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Only 3 encryption test cases was run. The test configuration is not correct. We should add all test case file in testconfiguration.properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9283:
Description:
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key. (Put in HIVE-9286)
was:
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key.
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key. (Put in HIVE-9286)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267361#comment-14267361
]
Dong Chen commented on HIVE-9283:
-
Sorry for typo... It is HIVE-9286
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267360#comment-14267360
]
Dong Chen commented on HIVE-9283:
-
Sorry for typo... It is HIVE-9286
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267357#comment-14267357
]
Dong Chen commented on HIVE-9283:
-
Good suggestion! This can make the piece of code focused, and easy to revert or
review. Thank [~Ferd]
I put the 2nd point in HIVE-9268.
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9286) Add clean up code in some encryption related test cases
Dong Chen created HIVE-9286: --- Summary: Add clean up code in some encryption related test cases Key: HIVE-9286 URL: https://issues.apache.org/jira/browse/HIVE-9286 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Add cleanup at the end of .q files. drop table and delete key. Otherwise, some cases will fail caused by cannot create existed key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9285) The move task doesn't work for inserting overwrite a local directory in test mode
[ https://issues.apache.org/jira/browse/HIVE-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9285: --- Attachment: HIVE-9285-encryption.patch > The move task doesn't work for inserting overwrite a local directory in test > mode > - > > Key: HIVE-9285 > URL: https://issues.apache.org/jira/browse/HIVE-9285 > Project: Hive > Issue Type: Sub-task >Affects Versions: encryption-branch >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: encryption-branch > > Attachments: HIVE-9285-encryption.patch > > > In the query test, the data warehouse is using the file system instead of the > hdfs. So the schema will be "file://" instead of "hdfs://" that results isSub > method doesn't function well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9285) The move task doesn't work for inserting overwrite a local directory in test mode
[ https://issues.apache.org/jira/browse/HIVE-9285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9285: --- Status: Patch Available (was: Open) > The move task doesn't work for inserting overwrite a local directory in test > mode > - > > Key: HIVE-9285 > URL: https://issues.apache.org/jira/browse/HIVE-9285 > Project: Hive > Issue Type: Sub-task >Affects Versions: encryption-branch >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: encryption-branch > > Attachments: HIVE-9285-encryption.patch > > > In the query test, the data warehouse is using the file system instead of the > hdfs. So the schema will be "file://" instead of "hdfs://" that results isSub > method doesn't function well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9283:
Description:
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key.
was:
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key.
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9285) The move task doesn't work for inserting overwrite a local directory in test mode
Ferdinand Xu created HIVE-9285: -- Summary: The move task doesn't work for inserting overwrite a local directory in test mode Key: HIVE-9285 URL: https://issues.apache.org/jira/browse/HIVE-9285 Project: Hive Issue Type: Sub-task Affects Versions: encryption-branch Reporter: Ferdinand Xu Assignee: Ferdinand Xu Fix For: encryption-branch In the query test, the data warehouse is using the file system instead of the hdfs. So the schema will be "file://" instead of "hdfs://" that results isSub method doesn't function well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9284) The hdfsEncryptionShim does not handle the relative path well based on hadoop 2.6
[ https://issues.apache.org/jira/browse/HIVE-9284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9284: --- Status: Patch Available (was: Open) > The hdfsEncryptionShim does not handle the relative path well based on hadoop > 2.6 > - > > Key: HIVE-9284 > URL: https://issues.apache.org/jira/browse/HIVE-9284 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: encryption-branch > > Attachments: HIVE-9284-encryption.patch > > > The method isPathEncrypted is not able to handle the relative path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9284) The hdfsEncryptionShim does not handle the relative path well based on hadoop 2.6
[ https://issues.apache.org/jira/browse/HIVE-9284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9284: --- Attachment: HIVE-9284-encryption.patch > The hdfsEncryptionShim does not handle the relative path well based on hadoop > 2.6 > - > > Key: HIVE-9284 > URL: https://issues.apache.org/jira/browse/HIVE-9284 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: encryption-branch > > Attachments: HIVE-9284-encryption.patch > > > The method isPathEncrypted is not able to handle the relative path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9284) The hdfsEncryptionShim does not handle the relative path well based on hadoop 2.6
[ https://issues.apache.org/jira/browse/HIVE-9284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267351#comment-14267351 ] Ferdinand Xu commented on HIVE-9284: Stack is as follows: 2015-01-07 12:50:38,565 ERROR [HiveServer2-Background-Pool: Thread-42]: exec.Task (SessionState.java:printError(857)) - Failed with exception Pathname target/warehouse/dest4.out from target/warehouse/dest4.out is not a valid DFS filename. java.lang.IllegalArgumentException: Pathname target/warehouse/dest4.out from target/warehouse/dest4.out is not a valid DFS filename. at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:196) at org.apache.hadoop.hdfs.DistributedFileSystem.getEZForPath(DistributedFileSystem.java:1906) at org.apache.hadoop.hdfs.client.HdfsAdmin.getEncryptionZoneForPath(HdfsAdmin.java:262) at org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.isPathEncrypted(Hadoop23Shims.java:1005) at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2418) at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:105) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:222) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1633) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1392) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1179) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1040) at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:145) at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:70) at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:197) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java: > The hdfsEncryptionShim does not handle the relative path well based on hadoop > 2.6 > - > > Key: HIVE-9284 > URL: https://issues.apache.org/jira/browse/HIVE-9284 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: encryption-branch > > > The method isPathEncrypted is not able to handle the relative path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9284) The hdfsEncryptionShim does not handle the relative path well based on hadoop 2.6
Ferdinand Xu created HIVE-9284: -- Summary: The hdfsEncryptionShim does not handle the relative path well based on hadoop 2.6 Key: HIVE-9284 URL: https://issues.apache.org/jira/browse/HIVE-9284 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu Fix For: encryption-branch The method isPathEncrypted is not able to handle the relative path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Building Hive-0.14 is failing because artifact pentaho-aggdesigner-algorithm-5.1.3-jhyde could not be resolved
Conjars (http://conjars.org/) has all of these libraries. Calcite’s pom uses it: true always warn conjars Conjars http://conjars.org/repo default Julian On Jan 6, 2015, at 3:45 PM, Ravi Prakash wrote: > Hi folks! > I'm still stuck on this issue. Hopefully people are back from their vacations > and can now reply. Resending. > > > On Monday, December 29, 2014 1:17 PM, Ravi Prakash > wrote: > > > Hi folks! > I'm still somewhat stuck on this issue. Could someone please guide me? > I had to manually download these jars and put them into my ~/.m2/repository > from > http://repo.spring.io/libs-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.3-jhyde/ > . (When I configured this repository, maven somehow still didn't find the > artifacts) > I had to do this for the following > artifacts:http://repo.spring.io/libs-release/org/pentaho/pentaho-aggdesigner/5.1.3-jhyde/pentaho-aggdesigner-5.1.3-jhyde.pomhttp://repo.spring.io/libs-release/net/hydromatic/linq4j/0.4/linq4j-0.4.jarhttp://repo.spring.io/libs-release/net/hydromatic/quidem/0.1.1/quidem-0.1.1.pom > Consequently I was able to build hive, however I don't believe this is a > kosher way of producing production binaries, so I am still waiting to hear > from you all. Is there another trusted repository I can try which has these > artifacts indexed? > > ThanksRavi > > > On Monday, December 22, 2014 2:14 PM, Ravi Prakash > wrote: > > > Hi! > Has anyone tried building Hive-0.14 from source? I'm using the tag for > release-0.14.0 https://github.com/apache/hive/releases/tag/release-0.14.0 > > The command I use is: mvn install -DskipTests -Phadoop-2 > -DcreateChecksum=true -Dtez.version=0.5.3 -Dcalcite.version=0.9.2-incubating > > The build fails for me with the following error:[ERROR] Failed to execute > goal on project hive-exec: Could not resolve dependencies for project > org.apache.hive:hive-exec:jar:0.14.0: The following artifacts could not be > resolved: org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde, > net.hydromatic:linq4j:jar:0.4, net.hydromatic:quidem:jar:0.1.1: Could not > find artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde in > nexus (http://localhost:8081/nexus/content/groups/public) -> [Help 1] > > This is a transitive dependency via the "calcite-0.9.2-incubating" artifact. > Is there a JIRA which someone can please point me to? It seems wrong that an > artifact with version "5.1.3-jhyde" is required to build Apache Hive, no > disrespect to Julian. Am I missing something? > ThanksRavi > > > > >
[jira] [Commented] (HIVE-8485) HMS on Oracle incompatibility
[
https://issues.apache.org/jira/browse/HIVE-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267326#comment-14267326
]
Hive QA commented on HIVE-8485:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690452/HIVE-8485.patch
{color:red}ERROR:{color} -1 due to 445 failed/errored test(s), 3315 tests
executed
*Failed tests:*
{noformat}
TestDbTxnManager - did not produce a TEST-*.xml file
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file
TestFilterHooks - did not produce a TEST-*.xml file
TestHCatLoader - did not produce a TEST-*.xml file
TestHCatStorer - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestHiveAuthorizationTaskFactory - did not produce a TEST-*.xml file
TestInitiator - did not produce a TEST-*.xml file
TestMacroSemanticAnalyzer - did not produce a TEST-*.xml file
TestPartitionNameWhitelistValidation - did not produce a TEST-*.xml file
TestSemanticAnalysis - did not produce a TEST-*.xml file
TestStorageBasedMetastoreAuthorizationReads - did not produce a TEST-*.xml file
TestStreaming - did not produce a TEST-*.xml file
TestUpdateDeleteSemanticAnalyzer - did not produce a TEST-*.xml file
TestWorker - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.initializationError
org.apache.hadoop.hive.cli.TestCliDriverMethods.testProcessSelectDatabase
org.apache.hadoop.hive.cli.TestCliDriverMethods.testQuit
org.apache.hadoop.hive.cli.TestCliDriverMethods.testRun
org.apache.hadoop.hive.cli.TestCliDriverMethods.testprocessInitFiles
org.apache.hadoop.hive.cli.TestCliSessionState.testgetDbName
org.apache.hadoop.hive.cli.TestCompareCliDriver.initializationError
org.apache.hadoop.hive.cli.TestContribCliDriver.initializationError
org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.initializationError
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_external_table_ppd
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_external_table_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries_prefix
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_storage_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_custom_key3
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_bulk
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_handler_snapshot
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_joins
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_join
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_pushdown
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_scan_params
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_single_sourced_multi_insert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats2
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats3
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_stats_empty_partition
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_timestamp
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_ppd_key_ranges
org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_cascade_dbdrop_hadoop20
org.apache.hadoop.hive.cli.TestHBaseNegativeCliDriver.testCliDriver_generatehfiles_require_family_path
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.initializationError
org.apache.hadoop.hive.cli.TestMinimrCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeCliDriver.initializationError
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.initializationError
org.apache.hadoop.hive.hooks.TestHs2Hooks.org.apache.hadoop.hive.hooks.TestHs2Hooks
org.apache.hadoop.hive.metastore.TestAdminUser.testCreateAdminNAddUser
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInEmbed.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInEmbed
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.testCreateRole
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.testDropRole
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuth
[jira] [Commented] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267323#comment-14267323
]
Ferdinand Xu commented on HIVE-9283:
Hi [~dongc], we can track these two points in two separate jiras by creating
two subtasks for this jira. Any thoughts?
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9282) hive could not able to integrate with spark
[ https://issues.apache.org/jira/browse/HIVE-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267316#comment-14267316 ] Chinna Rao Lalam commented on HIVE-9282: Hive on spark is not available in hive-0.12.0. Currently work is happening on spark branch of hive. Soon it will be merged to trunk. > hive could not able to integrate with spark > --- > > Key: HIVE-9282 > URL: https://issues.apache.org/jira/browse/HIVE-9282 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 0.12.0 > Environment: centOS 6.4 and hadoop-1.0.4 and hive-0.12.0 and > spark-0.8.0 >Reporter: suraj > Fix For: spark-branch > > Original Estimate: 12h > Remaining Estimate: 12h > > i have installed hadoop-1.0.4 and on top this i have installed everything by > just following this site : > https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started > hive-0.12.0 > and spark-0.8.0 > in that site they have mentioned that i have to install spark-1.2.x assembly > but i have installed spark as spark-0.8.0 > even i have compiled hive library using maven still i am getting issue that > wrong FS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9219) Investigate differences for auto join tests in explain after merge from trunk [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267305#comment-14267305
]
Chao commented on HIVE-9219:
[~xuefuz] Nope, the differences are expected. I think we can close this one now.
> Investigate differences for auto join tests in explain after merge from trunk
> [Spark Branch]
>
>
> Key: HIVE-9219
> URL: https://issues.apache.org/jira/browse/HIVE-9219
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Brock Noland
>Assignee: Chao
>
> {noformat}
> diff --git a/ql/src/test/results/clientpositive/spark/auto_join14.q.out
> b/ql/src/test/results/clientpositive/spark/auto_join14.q.out
> index cbca649..830314e 100644
> --- a/ql/src/test/results/clientpositive/spark/auto_join14.q.out
> +++ b/ql/src/test/results/clientpositive/spark/auto_join14.q.out
> @@ -38,9 +38,6 @@ STAGE PLANS:
> predicate: (key > 100) (type: boolean)
> Statistics: Num rows: 166 Data size: 1763 Basic stats:
> COMPLETE Column stats: NONE
> Spark HashTable Sink Operator
> - condition expressions:
> -0
> -1 {value}
>keys:
> 0 key (type: string)
> 1 key (type: string)
> @@ -62,9 +59,6 @@ STAGE PLANS:
> Map Join Operator
>condition map:
> Inner Join 0 to 1
> - condition expressions:
> -0 {key}
> -1 {value}
>keys:
> 0 key (type: string)
> 1 key (type: string)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9283) Improve encryption related test cases
Dong Chen created HIVE-9283:
---
Summary: Improve encryption related test cases
Key: HIVE-9283
URL: https://issues.apache.org/jira/browse/HIVE-9283
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9219) Investigate differences for auto join tests in explain after merge from trunk [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267299#comment-14267299
]
Xuefu Zhang commented on HIVE-9219:
---
[~csun], anything to be done here? If not, we just close this as "not a
problem" then.
> Investigate differences for auto join tests in explain after merge from trunk
> [Spark Branch]
>
>
> Key: HIVE-9219
> URL: https://issues.apache.org/jira/browse/HIVE-9219
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Brock Noland
>Assignee: Chao
>
> {noformat}
> diff --git a/ql/src/test/results/clientpositive/spark/auto_join14.q.out
> b/ql/src/test/results/clientpositive/spark/auto_join14.q.out
> index cbca649..830314e 100644
> --- a/ql/src/test/results/clientpositive/spark/auto_join14.q.out
> +++ b/ql/src/test/results/clientpositive/spark/auto_join14.q.out
> @@ -38,9 +38,6 @@ STAGE PLANS:
> predicate: (key > 100) (type: boolean)
> Statistics: Num rows: 166 Data size: 1763 Basic stats:
> COMPLETE Column stats: NONE
> Spark HashTable Sink Operator
> - condition expressions:
> -0
> -1 {value}
>keys:
> 0 key (type: string)
> 1 key (type: string)
> @@ -62,9 +59,6 @@ STAGE PLANS:
> Map Join Operator
>condition map:
> Inner Join 0 to 1
> - condition expressions:
> -0 {key}
> -1 {value}
>keys:
> 0 key (type: string)
> 1 key (type: string)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9243) Static Map in IOContext is not thread safe
[
https://issues.apache.org/jira/browse/HIVE-9243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9243:
--
Resolution: Fixed
Fix Version/s: 0.15.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks, Brock.
> Static Map in IOContext is not thread safe
> --
>
> Key: HIVE-9243
> URL: https://issues.apache.org/jira/browse/HIVE-9243
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.15.0
>Reporter: Brock Noland
>Assignee: Brock Noland
> Fix For: 0.15.0
>
> Attachments: HIVE-9243.patch, HIVE-9243.patch, HIVE-9243.patch
>
>
> This map can be accessed by multiple threads. We can either map it a
> {{ConcurrentHashMap}} or synchronize the calls to this class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9154) Cache pathToPartitionInfo in context aware record reader
[ https://issues.apache.org/jira/browse/HIVE-9154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9154: -- Resolution: Fixed Fix Version/s: (was: spark-branch) Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Jimmy. > Cache pathToPartitionInfo in context aware record reader > > > Key: HIVE-9154 > URL: https://issues.apache.org/jira/browse/HIVE-9154 > Project: Hive > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.15.0 > > Attachments: HIVE-9154.1-spark.patch, HIVE-9154.1-spark.patch, > HIVE-9154.2.patch, HIVE-9154.3.patch > > > This is similar to HIVE-9127. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9282) hive could not able to integrate with spark
suraj created HIVE-9282: --- Summary: hive could not able to integrate with spark Key: HIVE-9282 URL: https://issues.apache.org/jira/browse/HIVE-9282 Project: Hive Issue Type: Bug Components: Spark Affects Versions: 0.12.0 Environment: centOS 6.4 and hadoop-1.0.4 and hive-0.12.0 and spark-0.8.0 Reporter: suraj Fix For: spark-branch i have installed hadoop-1.0.4 and on top this i have installed everything by just following this site : https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started hive-0.12.0 and spark-0.8.0 in that site they have mentioned that i have to install spark-1.2.x assembly but i have installed spark as spark-0.8.0 even i have compiled hive library using maven still i am getting issue that wrong FS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267293#comment-14267293 ] Xuefu Zhang commented on HIVE-9251: --- I see it in the code now. Patch looks good. I just had one minor comment/question on RB. > SetSparkReducerParallelism is likely to set too small number of reducers > [Spark Branch] > --- > > Key: HIVE-9251 > URL: https://issues.apache.org/jira/browse/HIVE-9251 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch > > > This may hurt performance or even lead to task failures. For example, spark's > netty-based shuffle limits the max frame size to be 2G. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9281) Code cleanup [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9281: Attachment: HIVE-9281.patch > Code cleanup [Spark Branch] > --- > > Key: HIVE-9281 > URL: https://issues.apache.org/jira/browse/HIVE-9281 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-9281.patch > > > In preparation for merge, we need to cleanup the codes. > This includes removing TODO's, fixing checkstyles, removing commented or > unused code, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9281) Code cleanup [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9281: Attachment: (was: HIVE-9281.patch) > Code cleanup [Spark Branch] > --- > > Key: HIVE-9281 > URL: https://issues.apache.org/jira/browse/HIVE-9281 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-9281.patch > > > In preparation for merge, we need to cleanup the codes. > This includes removing TODO's, fixing checkstyles, removing commented or > unused code, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9281) Code cleanup [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9281: Status: Open (was: Patch Available) > Code cleanup [Spark Branch] > --- > > Key: HIVE-9281 > URL: https://issues.apache.org/jira/browse/HIVE-9281 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-9281.patch > > > In preparation for merge, we need to cleanup the codes. > This includes removing TODO's, fixing checkstyles, removing commented or > unused code, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9281) Code cleanup [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9281: Status: Patch Available (was: Open) > Code cleanup [Spark Branch] > --- > > Key: HIVE-9281 > URL: https://issues.apache.org/jira/browse/HIVE-9281 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-9281.patch > > > In preparation for merge, we need to cleanup the codes. > This includes removing TODO's, fixing checkstyles, removing commented or > unused code, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9281) Code cleanup [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267280#comment-14267280 ] Szehon Ho commented on HIVE-9281: - Note in regard to checkstyle fixes, I took the low-hanging fruit like code formatting, naming, but ignored some suggestion like put 'final' on all the parameters, methods as it would be too much. > Code cleanup [Spark Branch] > --- > > Key: HIVE-9281 > URL: https://issues.apache.org/jira/browse/HIVE-9281 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-9281.patch > > > In preparation for merge, we need to cleanup the codes. > This includes removing TODO's, fixing checkstyles, removing commented or > unused code, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9281) Code cleanup [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9281: Attachment: HIVE-9281.patch There's some issue attaching patch to the review board. Attaching a first pass here. It mostly fixes checkstyle warnings, and general code cleanup. It still leaves a lot of TODO's, I will need to follow up with individual authors and do follow-up JIRA's. > Code cleanup [Spark Branch] > --- > > Key: HIVE-9281 > URL: https://issues.apache.org/jira/browse/HIVE-9281 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > Attachments: HIVE-9281.patch > > > In preparation for merge, we need to cleanup the codes. > This includes removing TODO's, fixing checkstyles, removing commented or > unused code, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Parquet Branch
Hi, Some of us are looking at doing some substantial work on the parquet integration. We'd like to use a branch so we can commit experimental code. Unless there is an objection I will create a branch for this work. Cheers, Brock
[jira] [Commented] (HIVE-9195) CBO changes constant to column type
[
https://issues.apache.org/jira/browse/HIVE-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267277#comment-14267277
]
Navis commented on HIVE-9195:
-
[~ashutoshc] I see the case you've mentioned in union7, etc. and sound
reasonable to me. Make an issue for it and let's see what will happen.
> CBO changes constant to column type
> ---
>
> Key: HIVE-9195
> URL: https://issues.apache.org/jira/browse/HIVE-9195
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 0.14.0
>Reporter: Navis
>Assignee: Navis
> Fix For: 0.15.0
>
> Attachments: HIVE-9195.1.patch.txt, HIVE-9195.2.patch.txt,
> HIVE-9195.3.patch.txt
>
>
> Making testcase for HIVE-8613, I've found CBO changes constant expr to column
> expr. For example (only in test mode).
> {code}
> CREATE TABLE bucket (key double, value string) CLUSTERED BY (key) SORTED BY
> (key DESC) INTO 4 BUCKETS STORED AS TEXTFILE;
> load data local inpath '../../data/files/srcsortbucket1outof4.txt' INTO TABLE
> bucket;
> load data local inpath '../../data/files/srcsortbucket2outof4.txt' INTO TABLE
> bucket;
> load data local inpath '../../data/files/srcsortbucket3outof4.txt' INTO TABLE
> bucket;
> load data local inpath '../../data/files/srcsortbucket4outof4.txt' INTO TABLE
> bucket;
> select percentile_approx(case when key < 100 then cast('NaN' as double) else
> key end, 0.5) from bucket;
> {code}
> It works in shell but in TestCliDriver, that induces argument type exception
> creating udaf evaluator, which expects constant OI for second argument.
> {noformat}
> 2014-12-22 17:03:31,433 ERROR parse.SemanticAnalyzer
> (SemanticAnalyzer.java:analyzeInternal(10102)) - CBO failed, skipping CBO.
> org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: The second argument
> must be a constant, but double was passed instead.
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox.getEvaluator(GenericUDAFPercentileApprox.java:146)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1160)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:3794)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:4467)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:5536)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8884)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9745)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9638)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10086)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1107)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1155)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1034)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:206)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:158)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:369)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:304)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:877)
> at
> org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:136)
> at
> org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23(TestCliDriver.java:120)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267276#comment-14267276 ] Rui Li commented on HIVE-9251: -- That basically means cluster info is not available. So hive will decide number of reducers only by the input size. > SetSparkReducerParallelism is likely to set too small number of reducers > [Spark Branch] > --- > > Key: HIVE-9251 > URL: https://issues.apache.org/jira/browse/HIVE-9251 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch > > > This may hurt performance or even lead to task failures. For example, spark's > netty-based shuffle limits the max frame size to be 2G. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9281) Code cleanup [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szehon Ho updated HIVE-9281: Issue Type: Sub-task (was: Task) Parent: HIVE-7292 > Code cleanup [Spark Branch] > --- > > Key: HIVE-9281 > URL: https://issues.apache.org/jira/browse/HIVE-9281 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: spark-branch >Reporter: Szehon Ho >Assignee: Szehon Ho > > In preparation for merge, we need to cleanup the codes. > This includes removing TODO's, fixing checkstyles, removing commented or > unused code, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9281) Code cleanup [Spark Branch]
Szehon Ho created HIVE-9281: --- Summary: Code cleanup [Spark Branch] Key: HIVE-9281 URL: https://issues.apache.org/jira/browse/HIVE-9281 Project: Hive Issue Type: Task Components: Spark Affects Versions: spark-branch Reporter: Szehon Ho Assignee: Szehon Ho In preparation for merge, we need to cleanup the codes. This includes removing TODO's, fixing checkstyles, removing commented or unused code, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9278) Cached expression feature broken in one case
[
https://issues.apache.org/jira/browse/HIVE-9278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-9278:
Assignee: Navis
Status: Patch Available (was: Open)
> Cached expression feature broken in one case
>
>
> Key: HIVE-9278
> URL: https://issues.apache.org/jira/browse/HIVE-9278
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Navis
>Priority: Critical
> Attachments: HIVE-9278.1.patch.txt
>
>
> Different query result depending on whether hive.cache.expr.evaluation is
> true or false. When true, no query results are produced (this is wrong).
> The q file:
> {noformat}
> set hive.cache.expr.evaluation=true;
> CREATE TABLE cache_expr_repro (date_str STRING);
> LOAD DATA LOCAL INPATH '../../data/files/cache_expr_repro.txt' INTO TABLE
> cache_expr_repro;
> SELECT MONTH(date_str) AS `mon`, CAST((MONTH(date_str) - 1) / 3 + 1 AS int)
> AS `quarter`, YEAR(date_str) AS `year` FROM cache_expr_repro WHERE
> ((CAST((MONTH(date_str) - 1) / 3 + 1 AS int) = 1) AND (YEAR(date_str) =
> 2015)) GROUP BY MONTH(date_str), CAST((MONTH(date_str) - 1) / 3 + 1 AS int),
> YEAR(date_str) ;
> {noformat}
> cache_expr_repro.txt
> {noformat}
> 2015-01-01 00:00:00
> 2015-02-01 00:00:00
> 2015-01-01 00:00:00
> 2015-02-01 00:00:00
> 2015-01-01 00:00:00
> 2015-01-01 00:00:00
> 2015-02-01 00:00:00
> 2015-02-01 00:00:00
> 2015-01-01 00:00:00
> 2015-01-01 00:00:00
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9278) Cached expression feature broken in one case
[
https://issues.apache.org/jira/browse/HIVE-9278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-9278:
Attachment: HIVE-9278.1.patch.txt
My bad, again.
> Cached expression feature broken in one case
>
>
> Key: HIVE-9278
> URL: https://issues.apache.org/jira/browse/HIVE-9278
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Priority: Critical
> Attachments: HIVE-9278.1.patch.txt
>
>
> Different query result depending on whether hive.cache.expr.evaluation is
> true or false. When true, no query results are produced (this is wrong).
> The q file:
> {noformat}
> set hive.cache.expr.evaluation=true;
> CREATE TABLE cache_expr_repro (date_str STRING);
> LOAD DATA LOCAL INPATH '../../data/files/cache_expr_repro.txt' INTO TABLE
> cache_expr_repro;
> SELECT MONTH(date_str) AS `mon`, CAST((MONTH(date_str) - 1) / 3 + 1 AS int)
> AS `quarter`, YEAR(date_str) AS `year` FROM cache_expr_repro WHERE
> ((CAST((MONTH(date_str) - 1) / 3 + 1 AS int) = 1) AND (YEAR(date_str) =
> 2015)) GROUP BY MONTH(date_str), CAST((MONTH(date_str) - 1) / 3 + 1 AS int),
> YEAR(date_str) ;
> {noformat}
> cache_expr_repro.txt
> {noformat}
> 2015-01-01 00:00:00
> 2015-02-01 00:00:00
> 2015-01-01 00:00:00
> 2015-02-01 00:00:00
> 2015-01-01 00:00:00
> 2015-01-01 00:00:00
> 2015-02-01 00:00:00
> 2015-02-01 00:00:00
> 2015-01-01 00:00:00
> 2015-01-01 00:00:00
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-8578) Investigate test failures related to HIVE-8545 [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang resolved HIVE-8578.
---
Resolution: Cannot Reproduce
Ok, let's close it now. We can re-open it or file a new issue if the problem
happens again.
> Investigate test failures related to HIVE-8545 [Spark Branch]
> -
>
> Key: HIVE-8578
> URL: https://issues.apache.org/jira/browse/HIVE-8578
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Chao
>Assignee: Jimmy Xiang
>
> In HIVE-8545, there are a few test failures, for instance,
> {{multi_insert_lateral_view.q}} and {{ppd_multi_insert.q}}. They appear to be
> happening at random, and not reproducible locally. We need to track down the
> root cause, and fix in this JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9249) Vectorization: Join involving CHAR/VARCHAR fails during execution (java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-9249:
---
Description:
VectorColumnAssignFactory doesn't handle HiveCharWritable / HiveVarcharWritable
objects.
Either:
HiveVarcharWritable cannot be cast to ... HiveVarchar
or
HiveCharWritable cannot be cast to ... HiveChar
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 24 more
{code}
was:
VectorColumnAssignFactory doesn't handle HiveCharWritable / HiveVarcharWritable
objects.
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 24 more
{code}
> Vectorization: Join involving CHAR/VARCHAR fails during execution
> (java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.hive.common.type.HiveVarchar when joining tables)
> ---
>
> Key: HIVE-9249
> URL: https://issues.apache.org/jira/browse/HIVE-9249
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-9249.01.patch
>
>
> VectorColumnAssignFactory doesn't handle HiveCharWritable /
> HiveVarcharWritable objects.
> Either:
> HiveVarcharWritable cannot be cast to ... HiveVarchar
> or
> HiveCharWritable cannot be cast to ... HiveChar
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.hive.common.type.HiveVarchar
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9249) Vectorization: Join involving CHAR/VARCHAR fails during execution (java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-9249:
---
Description:
VectorColumnAssignFactory doesn't handle HiveCharWritable / HiveVarcharWritable
objects.
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 24 more
{code}
was:
VectorColumnAssignFactory doesn't handle HiveCharWritable / HiveVarcharWritable
objects.
{code}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar
at
org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
at
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
... 24 more
{code}
> Vectorization: Join involving CHAR/VARCHAR fails during execution
> (java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.hive.common.type.HiveVarchar when joining tables)
> ---
>
> Key: HIVE-9249
> URL: https://issues.apache.org/jira/browse/HIVE-9249
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-9249.01.patch
>
>
> VectorColumnAssignFactory doesn't handle HiveCharWritable /
> HiveVarcharWritable objects.
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.hive.common.type.HiveVarchar
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9249) Vectorization: Join involving CHAR/VARCHAR fails during execution (java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to org
[
https://issues.apache.org/jira/browse/HIVE-9249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-9249:
---
Summary: Vectorization: Join involving CHAR/VARCHAR fails during execution
(java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
org.apache.hadoop.hive.common.type.HiveVarchar when joining tables) (was:
Vectorization: Join involving CHAR/VARCHAR fails during execution)
> Vectorization: Join involving CHAR/VARCHAR fails during execution
> (java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.hive.common.type.HiveVarchar when joining tables)
> ---
>
> Key: HIVE-9249
> URL: https://issues.apache.org/jira/browse/HIVE-9249
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Affects Versions: 0.14.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-9249.01.patch
>
>
> VectorColumnAssignFactory doesn't handle HiveCharWritable /
> HiveVarcharWritable objects.
> {code}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.io.HiveVarcharWritable cannot be cast to
> org.apache.hadoop.hive.common.type.HiveVarchar
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorColumnAssignFactory$17.assignObjectValue(VectorColumnAssignFactory.java:417)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.internalForward(VectorMapJoinOperator.java:196)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:748)
> at
> org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:299)
> ... 24 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267243#comment-14267243 ] Xuefu Zhang commented on HIVE-9251: --- The patch looks good. One question though: (-1, -1) is returned for get memory and core call, which makes me wonder what's the behavior on Hive side if that's the case. Should we somehow safeguard on this? > SetSparkReducerParallelism is likely to set too small number of reducers > [Spark Branch] > --- > > Key: HIVE-9251 > URL: https://issues.apache.org/jira/browse/HIVE-9251 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch > > > This may hurt performance or even lead to task failures. For example, spark's > netty-based shuffle limits the max frame size to be 2G. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9280) The Arguments of CRYPTO command is not parsed correctly in QTestUtil.executeTestCommand()
[ https://issues.apache.org/jira/browse/HIVE-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9280: Attachment: HIVE-9280.patch A small fix. The command might start with \n, so substring it without trim first might index incorrectly. > The Arguments of CRYPTO command is not parsed correctly in > QTestUtil.executeTestCommand() > - > > Key: HIVE-9280 > URL: https://issues.apache.org/jira/browse/HIVE-9280 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9280.patch > > > NO PRECOMMIT TESTS > The Arguments of CRYPTO command is not parsed correctly in > QTestUtil.executeTestCommand(). It is found after fix HIVE-9279. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267229#comment-14267229 ] Rui Li commented on HIVE-9251: -- Hi [~xuefuz], yeah I'll update the golden files if you think the patch is OK. > SetSparkReducerParallelism is likely to set too small number of reducers > [Spark Branch] > --- > > Key: HIVE-9251 > URL: https://issues.apache.org/jira/browse/HIVE-9251 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch > > > This may hurt performance or even lead to task failures. For example, spark's > netty-based shuffle limits the max frame size to be 2G. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9280) The Arguments of CRYPTO command is not parsed correctly in QTestUtil.executeTestCommand()
[ https://issues.apache.org/jira/browse/HIVE-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9280: Status: Patch Available (was: Open) > The Arguments of CRYPTO command is not parsed correctly in > QTestUtil.executeTestCommand() > - > > Key: HIVE-9280 > URL: https://issues.apache.org/jira/browse/HIVE-9280 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > > NO PRECOMMIT TESTS > The Arguments of CRYPTO command is not parsed correctly in > QTestUtil.executeTestCommand(). It is found after fix HIVE-9279. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267223#comment-14267223 ] Xuefu Zhang commented on HIVE-9251: --- Hi Rui, for our unit test, the input size and cluster are all fixed. It shouldn't matter whether reducer count is exposed in the plan. As to the question of whether or not, we briefly discussed about this today and we will try to use the same RSC with query execution for explain query. If this can be nicely shared, it seems okay to have it in the plan. Let me know if I missed anything. > SetSparkReducerParallelism is likely to set too small number of reducers > [Spark Branch] > --- > > Key: HIVE-9251 > URL: https://issues.apache.org/jira/browse/HIVE-9251 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch > > > This may hurt performance or even lead to task failures. For example, spark's > netty-based shuffle limits the max frame size to be 2G. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9280) The Arguments of CRYPTO command is not parsed correctly in QTestUtil.executeTestCommand()
Dong Chen created HIVE-9280: --- Summary: The Arguments of CRYPTO command is not parsed correctly in QTestUtil.executeTestCommand() Key: HIVE-9280 URL: https://issues.apache.org/jira/browse/HIVE-9280 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen NO PRECOMMIT TESTS The Arguments of CRYPTO command is not parsed correctly in QTestUtil.executeTestCommand(). It is found after fix HIVE-9279. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9251) SetSparkReducerParallelism is likely to set too small number of reducers [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267213#comment-14267213 ] Rui Li commented on HIVE-9251: -- I quickly checked the failed tests. Most of them are in query plan because number of reducers changed. Some may also need a SORT_QUERY_RESULT tag. If we want to decide number of reducers based on input size and cluster info, maybe we shouldn't expose it in the query plan, given that input size may change and we currently need some hacks/workarounds to get spark cluster info. Any ideas? > SetSparkReducerParallelism is likely to set too small number of reducers > [Spark Branch] > --- > > Key: HIVE-9251 > URL: https://issues.apache.org/jira/browse/HIVE-9251 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-9251.1-spark.patch, HIVE-9251.2-spark.patch > > > This may hurt performance or even lead to task failures. For example, spark's > netty-based shuffle limits the max frame size to be 2G. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9279) TestEncryptedHDFSCliDriver get exception "Could not execute test command" for encryption test cases
[
https://issues.apache.org/jira/browse/HIVE-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9279:
Attachment: HIVE-9279.patch
Uploaded patch. Should trim the command string to get rid of '\n'.
> TestEncryptedHDFSCliDriver get exception "Could not execute test command" for
> encryption test cases
> ---
>
> Key: HIVE-9279
> URL: https://issues.apache.org/jira/browse/HIVE-9279
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>Priority: Blocker
> Fix For: encryption-branch
>
> Attachments: HIVE-9279.patch
>
>
> NO PRECOMMIT TESTS
> TestEncryptedHDFSCliDriver get exception "Could not execute test command" for
> encryption test cases.
> The stacktrace is:
> {quote}
> Unexpected exception java.lang.RuntimeException: Could not execute test
> command: Could not get CommandProcessor for command:
> at org.apache.hadoop.hive.ql.QTestUtil.executeTestCommand(QTestUtil.java:967)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:931)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:922)
> at
> org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.runTest(TestEncryptedHDFSCliDriver.java:150)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9279) TestEncryptedHDFSCliDriver get exception "Could not execute test command" for encryption test cases
[
https://issues.apache.org/jira/browse/HIVE-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9279:
Fix Version/s: encryption-branch
Status: Patch Available (was: Open)
> TestEncryptedHDFSCliDriver get exception "Could not execute test command" for
> encryption test cases
> ---
>
> Key: HIVE-9279
> URL: https://issues.apache.org/jira/browse/HIVE-9279
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>Priority: Blocker
> Fix For: encryption-branch
>
>
> NO PRECOMMIT TESTS
> TestEncryptedHDFSCliDriver get exception "Could not execute test command" for
> encryption test cases.
> The stacktrace is:
> {quote}
> Unexpected exception java.lang.RuntimeException: Could not execute test
> command: Could not get CommandProcessor for command:
> at org.apache.hadoop.hive.ql.QTestUtil.executeTestCommand(QTestUtil.java:967)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:931)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:922)
> at
> org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.runTest(TestEncryptedHDFSCliDriver.java:150)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9279) TestEncryptedHDFSCliDriver get exception "Could not execute test command" for encryption test cases
Dong Chen created HIVE-9279:
---
Summary: TestEncryptedHDFSCliDriver get exception "Could not
execute test command" for encryption test cases
Key: HIVE-9279
URL: https://issues.apache.org/jira/browse/HIVE-9279
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Priority: Blocker
NO PRECOMMIT TESTS
TestEncryptedHDFSCliDriver get exception "Could not execute test command" for
encryption test cases.
The stacktrace is:
{quote}
Unexpected exception java.lang.RuntimeException: Could not execute test
command: Could not get CommandProcessor for command:
at org.apache.hadoop.hive.ql.QTestUtil.executeTestCommand(QTestUtil.java:967)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:931)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:922)
at
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.runTest(TestEncryptedHDFSCliDriver.java:150)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-4841) Add partition level hook to HiveMetaHook
[ https://issues.apache.org/jira/browse/HIVE-4841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267192#comment-14267192 ] Navis commented on HIVE-4841: - Except TestFileSinkOperator, other failures would be fixed with clean build of metastore (return type changes are not detected by compiler). > Add partition level hook to HiveMetaHook > > > Key: HIVE-4841 > URL: https://issues.apache.org/jira/browse/HIVE-4841 > Project: Hive > Issue Type: Improvement > Components: StorageHandler >Reporter: Navis >Assignee: Navis >Priority: Minor > Attachments: HIVE-4841.4.patch.txt, HIVE-4841.D11673.1.patch, > HIVE-4841.D11673.2.patch, HIVE-4841.D11673.3.patch > > > Current HiveMetaHook provides hooks for tables only. With partition level > hook, external storages also could be revised to exploit PPR. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-6679: - Labels: TODOC14 TODOC15 (was: TODOC14) > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0 >Reporter: Prasad Mujumdar >Assignee: Navis > Labels: TODOC14, TODOC15 > Fix For: 0.15.0, 0.14.1 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch > > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267186#comment-14267186 ] Lefty Leverenz commented on HIVE-6679: -- bq. I'll create a new patch for trunk with just the socket timeout code. Does this mean *hive.server2.tcp.socket.keepalive* will only exist in 0.14.1, not in 0.15? > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0 >Reporter: Prasad Mujumdar >Assignee: Navis > Labels: TODOC14, TODOC15 > Fix For: 0.15.0, 0.14.1 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch > > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9217: -- Attachment: HIVE-9217.03.patch > UnionProcessor misses results for multi-insert when > hive.optimize.union.remove=true > --- > > Key: HIVE-9217 > URL: https://issues.apache.org/jira/browse/HIVE-9217 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch, > HIVE-9217.03.patch > > > To reproduce: > In union_remove_6.q, just change > FROM ( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > to > FROM ( > select * from( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > )subq > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9217: -- Status: Patch Available (was: Open) > UnionProcessor misses results for multi-insert when > hive.optimize.union.remove=true > --- > > Key: HIVE-9217 > URL: https://issues.apache.org/jira/browse/HIVE-9217 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch, > HIVE-9217.03.patch > > > To reproduce: > In union_remove_6.q, just change > FROM ( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > to > FROM ( > select * from( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > )subq > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9217: -- Attachment: (was: HIVE-9217.03.patch) > UnionProcessor misses results for multi-insert when > hive.optimize.union.remove=true > --- > > Key: HIVE-9217 > URL: https://issues.apache.org/jira/browse/HIVE-9217 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch, > HIVE-9217.03.patch > > > To reproduce: > In union_remove_6.q, just change > FROM ( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > to > FROM ( > select * from( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > )subq > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9217: -- Status: Open (was: Patch Available) > UnionProcessor misses results for multi-insert when > hive.optimize.union.remove=true > --- > > Key: HIVE-9217 > URL: https://issues.apache.org/jira/browse/HIVE-9217 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch, > HIVE-9217.03.patch > > > To reproduce: > In union_remove_6.q, just change > FROM ( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > to > FROM ( > select * from( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > )subq > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-4639) Add has null flag to ORC internal index
[
https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267182#comment-14267182
]
Hive QA commented on HIVE-4639:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690444/HIVE-4639.1.patch
{color:red}ERROR:{color} -1 due to 32 failed/errored test(s), 6731 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_full
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_extrapolate_part_stats_partial
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_predicate_pushdown
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_opt_vectorization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_analyze
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testCombinationInputFormatWithAcid
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.test1[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.test1[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testReadFormat_0_11[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testReadFormat_0_11[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testStringAndBinaryStatistics[0]
org.apache.hadoop.hive.ql.io.orc.TestOrcFile.testStringAndBinaryStatistics[1]
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testColumnsWithNullAndCompression
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testMultiStripeWithNull
org.apache.hadoop.hive.ql.io.orc.TestOrcNullOptimization.testMultiStripeWithoutNull
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testOrcSerDeStatsComplex
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testOrcSerDeStatsComplexOldFormat
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testSerdeStatsOldFormat
org.apache.hadoop.hive.ql.io.orc.TestOrcSerDeStats.testStringAndBinaryStatistics
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2274/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2274/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2274/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 32 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690444 - PreCommit-HIVE-TRUNK-Build
> Add has null flag to ORC internal index
> ---
>
> Key: HIVE-4639
> URL: https://issues.apache.org/jira/browse/HIVE-4639
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Reporter: Owen O'Malley
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-4639.1.patch
>
>
> It would enable more predicate pushdown if we added a flag to the index entry
> recording if there were any null values in the column for the 10k rows.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 29439: UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29439/ --- (Updated Jan. 7, 2015, 2:12 a.m.) Review request for hive and John Pullokkaran. Changes --- add notes and test files according to John's comments. Repository: hive-git Description --- In union_remove_6.q, just change FROM ( SELECT key, count(1) as values from inputTbl1 group by key UNION ALL SELECT key, count(1) as values from inputTbl1 group by key ) a insert overwrite table outputTbl1 select * insert overwrite table outputTbl2 select *; to FROM ( select * from( SELECT key, count(1) as values from inputTbl1 group by key UNION ALL SELECT key, count(1) as values from inputTbl1 group by key )subq ) a insert overwrite table outputTbl1 select * insert overwrite table outputTbl2 select *; select * from outputtbl2 will output nothing Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java a985c4f ql/src/test/queries/clientpositive/union_remove_6_subq.q PRE-CREATION ql/src/test/results/clientpositive/union_remove_6_subq.q.out PRE-CREATION Diff: https://reviews.apache.org/r/29439/diff/ Testing --- Thanks, pengcheng xiong
[jira] [Updated] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9217: -- Status: Patch Available (was: Open) > UnionProcessor misses results for multi-insert when > hive.optimize.union.remove=true > --- > > Key: HIVE-9217 > URL: https://issues.apache.org/jira/browse/HIVE-9217 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch, > HIVE-9217.03.patch > > > To reproduce: > In union_remove_6.q, just change > FROM ( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > to > FROM ( > select * from( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > )subq > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9217: -- Attachment: HIVE-9217.03.patch add notes and test files according to [~jpullokkaran]'s comments. > UnionProcessor misses results for multi-insert when > hive.optimize.union.remove=true > --- > > Key: HIVE-9217 > URL: https://issues.apache.org/jira/browse/HIVE-9217 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch, > HIVE-9217.03.patch > > > To reproduce: > In union_remove_6.q, just change > FROM ( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > to > FROM ( > select * from( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > )subq > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9217) UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
[ https://issues.apache.org/jira/browse/HIVE-9217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9217: -- Status: Open (was: Patch Available) > UnionProcessor misses results for multi-insert when > hive.optimize.union.remove=true > --- > > Key: HIVE-9217 > URL: https://issues.apache.org/jira/browse/HIVE-9217 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-9217.01.patch, HIVE-9217.02.patch > > > To reproduce: > In union_remove_6.q, just change > FROM ( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > to > FROM ( > select * from( > SELECT key, count(1) as values from inputTbl1 group by key > UNION ALL > SELECT key, count(1) as values from inputTbl1 group by key > )subq > ) a > insert overwrite table outputTbl1 select * > insert overwrite table outputTbl2 select *; > select * from outputtbl2 will output nothing -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-6679: - Labels: TODOC14 (was: ) > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0 >Reporter: Prasad Mujumdar >Assignee: Navis > Labels: TODOC14 > Fix For: 0.15.0, 0.14.1 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch > > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-3972) Support using multiple reducer for fetching order by results
[
https://issues.apache.org/jira/browse/HIVE-3972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-3972:
Attachment: HIVE-3972.10.patch.txt
> Support using multiple reducer for fetching order by results
>
>
> Key: HIVE-3972
> URL: https://issues.apache.org/jira/browse/HIVE-3972
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D8349.5.patch, D8349.6.patch, D8349.7.patch,
> HIVE-3972.10.patch.txt, HIVE-3972.8.patch.txt, HIVE-3972.9.patch.txt,
> HIVE-3972.D8349.1.patch, HIVE-3972.D8349.2.patch, HIVE-3972.D8349.3.patch,
> HIVE-3972.D8349.4.patch
>
>
> Queries for fetching results which have lastly "order by" clause make final
> MR run with single reducer, which can be too much. For example,
> {code}
> select value, sum(key) as sum from src group by value order by sum;
> {code}
> If number of reducer is reasonable, multiple result files could be merged
> into single sorted stream in the fetcher level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4790) MapredLocalTask task does not make virtual columns
[
https://issues.apache.org/jira/browse/HIVE-4790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navis updated HIVE-4790:
Attachment: HIVE-4790.12.patch.txt
> MapredLocalTask task does not make virtual columns
> --
>
> Key: HIVE-4790
> URL: https://issues.apache.org/jira/browse/HIVE-4790
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Reporter: Navis
>Assignee: Navis
>Priority: Minor
> Attachments: D11511.3.patch, D11511.4.patch, HIVE-4790.10.patch.txt,
> HIVE-4790.11.patch.txt, HIVE-4790.12.patch.txt, HIVE-4790.5.patch.txt,
> HIVE-4790.6.patch.txt, HIVE-4790.7.patch.txt, HIVE-4790.8.patch.txt,
> HIVE-4790.9.patch.txt, HIVE-4790.D11511.1.patch, HIVE-4790.D11511.2.patch
>
>
> From mailing list,
> http://www.mail-archive.com/user@hive.apache.org/msg08264.html
> {noformat}
> SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON
> b.rownumber = a.number;
> fails with this error:
>
> > SELECT *,b.BLOCK__OFFSET__INSIDE__FILE FROM a JOIN b ON b.rownumber =
> a.number;
> Automatically selecting local only mode for query
> Total MapReduce jobs = 1
> setting HADOOP_USER_NAMEpmarron
> 13/06/25 10:52:56 WARN conf.HiveConf: DEPRECATED: Configuration property
> hive.metastore.local no longer has any effect. Make sure to provide a valid
> value for hive.metastore.uris if you are connecting to a remote metastore.
> Execution log at: /tmp/pmarron/.log
> 2013-06-25 10:52:56 Starting to launch local task to process map join;
> maximum memory = 932118528
> java.lang.RuntimeException: cannot find field block__offset__inside__file
> from [0:rownumber, 1:offset]
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.getStandardStructFieldRef(ObjectInspectorUtils.java:366)
> at
> org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector.getStructFieldRef(LazySimpleStructObjectInspector.java:168)
> at
> org.apache.hadoop.hive.serde2.objectinspector.DelegatedStructObjectInspector.getStructFieldRef(DelegatedStructObjectInspector.java:74)
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:68)
> at
> org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:222)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:451)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:407)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:186)
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
> at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.initializeOperators(MapredLocalTask.java:394)
> at
> org.apache.hadoop.hive.ql.exec.MapredLocalTask.executeFromChildJVM(MapredLocalTask.java:277)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:676)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Execution failed with exit status: 2
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9278) Cached expression feature broken in one case
Matt McCline created HIVE-9278:
--
Summary: Cached expression feature broken in one case
Key: HIVE-9278
URL: https://issues.apache.org/jira/browse/HIVE-9278
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0
Reporter: Matt McCline
Priority: Critical
Different query result depending on whether hive.cache.expr.evaluation is true
or false. When true, no query results are produced (this is wrong).
The q file:
{noformat}
set hive.cache.expr.evaluation=true;
CREATE TABLE cache_expr_repro (date_str STRING);
LOAD DATA LOCAL INPATH '../../data/files/cache_expr_repro.txt' INTO TABLE
cache_expr_repro;
SELECT MONTH(date_str) AS `mon`, CAST((MONTH(date_str) - 1) / 3 + 1 AS int) AS
`quarter`, YEAR(date_str) AS `year` FROM cache_expr_repro WHERE
((CAST((MONTH(date_str) - 1) / 3 + 1 AS int) = 1) AND (YEAR(date_str) = 2015))
GROUP BY MONTH(date_str), CAST((MONTH(date_str) - 1) / 3 + 1 AS int),
YEAR(date_str) ;
{noformat}
cache_expr_repro.txt
{noformat}
2015-01-01 00:00:00
2015-02-01 00:00:00
2015-01-01 00:00:00
2015-02-01 00:00:00
2015-01-01 00:00:00
2015-01-01 00:00:00
2015-02-01 00:00:00
2015-02-01 00:00:00
2015-01-01 00:00:00
2015-01-01 00:00:00
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9277) Hybrid Hybrid Grace Hash Join
Wei Zheng created HIVE-9277: --- Summary: Hybrid Hybrid Grace Hash Join Key: HIVE-9277 URL: https://issues.apache.org/jira/browse/HIVE-9277 Project: Hive Issue Type: New Feature Components: Physical Optimizer Reporter: Wei Zheng We are proposing an enhanced hash join algorithm called “hybrid hybrid grace hash join”. We can benefit from this feature as illustrated below: o The query will not fail even if the estimated memory requirement is slightly wrong o Expensive garbage collection overhead can be avoided when hash table grows o Join execution using a Map join operator even though the small table doesn't fit in memory as spilling some data from the build and probe sides will still be cheaper than having to shuffle the large fact table The design was based on Hadoop’s parallel processing capability and significant amount of memory available. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-6173) Beeline doesn't accept --hiveconf option as Hive CLI does
[
https://issues.apache.org/jira/browse/HIVE-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-6173:
-
Comment: was deleted
(was:
Oops, my posting to the SQLLine mailing list didn't go through. So I'll try
again after my request to join the list is approved.)
> Beeline doesn't accept --hiveconf option as Hive CLI does
> -
>
> Key: HIVE-6173
> URL: https://issues.apache.org/jira/browse/HIVE-6173
> Project: Hive
> Issue Type: Improvement
> Components: CLI
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Labels: TODOC13
> Fix For: 0.13.0
>
> Attachments: HIVE-6173.1.patch, HIVE-6173.2.patch, HIVE-6173.patch
>
>
> {code}
> > beeline -u jdbc:hive2:// --hiveconf a=b
> Usage: java org.apache.hive.cli.beeline.BeeLine
> {code}
> Since Beeline is replacing Hive CLI, it should support this command line
> option as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9276) Exclude encryption related cases from TestCliDriver
[ https://issues.apache.org/jira/browse/HIVE-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9276: --- Attachment: HIVE-9276-encryption.patch > Exclude encryption related cases from TestCliDriver > --- > > Key: HIVE-9276 > URL: https://issues.apache.org/jira/browse/HIVE-9276 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: encryption-branch > > Attachments: HIVE-9276-encryption.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9276) Exclude encryption related cases from TestCliDriver
[ https://issues.apache.org/jira/browse/HIVE-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9276: --- Fix Version/s: encryption-branch > Exclude encryption related cases from TestCliDriver > --- > > Key: HIVE-9276 > URL: https://issues.apache.org/jira/browse/HIVE-9276 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > Fix For: encryption-branch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9276) Exclude encryption related cases from TestCliDriver
[ https://issues.apache.org/jira/browse/HIVE-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-9276: --- Status: Patch Available (was: Open) > Exclude encryption related cases from TestCliDriver > --- > > Key: HIVE-9276 > URL: https://issues.apache.org/jira/browse/HIVE-9276 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Ferdinand Xu > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9276) Exclude encryption related cases from TestCliDriver
Ferdinand Xu created HIVE-9276: -- Summary: Exclude encryption related cases from TestCliDriver Key: HIVE-9276 URL: https://issues.apache.org/jira/browse/HIVE-9276 Project: Hive Issue Type: Sub-task Reporter: Ferdinand Xu Assignee: Ferdinand Xu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8134) Umbrella JIRA to track concurrency improvements
[ https://issues.apache.org/jira/browse/HIVE-8134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu updated HIVE-8134: --- Sprint: Sprint - Jan - Mar > Umbrella JIRA to track concurrency improvements > --- > > Key: HIVE-8134 > URL: https://issues.apache.org/jira/browse/HIVE-8134 > Project: Hive > Issue Type: Improvement >Reporter: Brock Noland >Assignee: Ferdinand Xu > > The goal of this JIRA is track supportability issues with concurrent users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-9275) HIVE-9252
[ https://issues.apache.org/jira/browse/HIVE-9275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdinand Xu resolved HIVE-9275. Resolution: Invalid > HIVE-9252 > - > > Key: HIVE-9275 > URL: https://issues.apache.org/jira/browse/HIVE-9275 > Project: Hive > Issue Type: Bug >Reporter: Ferdinand Xu > Labels: hive-scrum > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8485) HMS on Oracle incompatibility
[
https://issues.apache.org/jira/browse/HIVE-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-8485:
---
Status: Patch Available (was: Open)
> HMS on Oracle incompatibility
> -
>
> Key: HIVE-8485
> URL: https://issues.apache.org/jira/browse/HIVE-8485
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Environment: Oracle as metastore DB
>Reporter: Ryan Pridgeon
>Assignee: Chaoyu Tang
> Attachments: HIVE-8485.patch
>
>
> Oracle does not distinguish between empty strings and NULL,which proves
> problematic for DataNucleus.
> In the event a user creates a table with some property stored as an empty
> string the table will no longer be accessible.
> i.e. TBLPROPERTIES ('serialization.null.format'='')
> If they try to select, describe, drop, etc the client prints the following
> exception.
> ERROR ql.Driver: FAILED: SemanticException [Error 10001]: Table not found
>
> The work around for this was to go into the hive metastore on the Oracle
> database and replace NULL with some other string. Users could then drop the
> tables or alter their data to use the new null format they just set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8485) HMS on Oracle incompatibility
[
https://issues.apache.org/jira/browse/HIVE-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-8485:
---
Attachment: HIVE-8485.patch
(Attaching patch)
> HMS on Oracle incompatibility
> -
>
> Key: HIVE-8485
> URL: https://issues.apache.org/jira/browse/HIVE-8485
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Environment: Oracle as metastore DB
>Reporter: Ryan Pridgeon
>Assignee: Chaoyu Tang
> Attachments: HIVE-8485.patch
>
>
> Oracle does not distinguish between empty strings and NULL,which proves
> problematic for DataNucleus.
> In the event a user creates a table with some property stored as an empty
> string the table will no longer be accessible.
> i.e. TBLPROPERTIES ('serialization.null.format'='')
> If they try to select, describe, drop, etc the client prints the following
> exception.
> ERROR ql.Driver: FAILED: SemanticException [Error 10001]: Table not found
>
> The work around for this was to go into the hive metastore on the Oracle
> database and replace NULL with some other string. Users could then drop the
> tables or alter their data to use the new null format they just set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8485) HMS on Oracle incompatibility
[
https://issues.apache.org/jira/browse/HIVE-8485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267083#comment-14267083
]
Sushanth Sowmyan commented on HIVE-8485:
I've done some debugging, and am able to verify the same issue. That said,
since thrift does not support nulls in maps, we should be verifying on our end
that we do not send back nulls in any case. To this end, I have a patch that
does the following:
* Introduces a new HiveConf parameter :
*hive.metastore.orm.retrieveMapNullsAsEmptyStrings* that defaults to *false*
* For operation with oracle, or any other datastore that convers empty
strings to nulls, we'd set this to true, but by default, it'd be false.
* If this parameter is set to true, ObjectStore will transform any null map
values it finds in any of the parameters maps to "".
* If this parameter is set to false, ObjectStore will prune any map entry
where the value is null.
This way, irrespective of whether or not we get nulls back from DN, we
sanity-guard our usage.
> HMS on Oracle incompatibility
> -
>
> Key: HIVE-8485
> URL: https://issues.apache.org/jira/browse/HIVE-8485
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Environment: Oracle as metastore DB
>Reporter: Ryan Pridgeon
>Assignee: Chaoyu Tang
>
> Oracle does not distinguish between empty strings and NULL,which proves
> problematic for DataNucleus.
> In the event a user creates a table with some property stored as an empty
> string the table will no longer be accessible.
> i.e. TBLPROPERTIES ('serialization.null.format'='')
> If they try to select, describe, drop, etc the client prints the following
> exception.
> ERROR ql.Driver: FAILED: SemanticException [Error 10001]: Table not found
>
> The work around for this was to go into the hive metastore on the Oracle
> database and replace NULL with some other string. Users could then drop the
> tables or alter their data to use the new null format they just set.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9275) HIVE-9252
Ferdinand Xu created HIVE-9275: -- Summary: HIVE-9252 Key: HIVE-9275 URL: https://issues.apache.org/jira/browse/HIVE-9275 Project: Hive Issue Type: Bug Reporter: Ferdinand Xu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9274) STORED BY not setting input and output formats correctly
Thomas Friedrich created HIVE-9274:
--
Summary: STORED BY not setting input and output formats correctly
Key: HIVE-9274
URL: https://issues.apache.org/jira/browse/HIVE-9274
Project: Hive
Issue Type: Bug
Affects Versions: 0.14.0
Reporter: Thomas Friedrich
When creating a new table using a storage handler via the STORED BY clause, for
example using the HBaseStorageHandler, the input and output formats are set to
null instead of the correct formats:
CREATE TABLE hbase_table_1(key int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:val")
TBLPROPERTIES ("hbase.table.name" = "xyz");
describe formatted hbase_table_1:
# Storage Information
SerDe Library: org.apache.hadoop.hive.hbase.HBaseSerDe
InputFormat:null
OutputFormat: null
In older Hive versions, it would set the correct formats.
Could it be related to the changes in HIVE-5976?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6173) Beeline doesn't accept --hiveconf option as Hive CLI does
[
https://issues.apache.org/jira/browse/HIVE-6173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267045#comment-14267045
]
Lefty Leverenz commented on HIVE-6173:
--
I posted a question on the SQLLine mailing list (sqlline-...@googlegroups.com)
asking for help with these doc issues.
By the way, four of the undocumented properties are in SqlLineOpts.java, but
_allowMultiLineCommand_ is only in BeeLineOpts.java.
[~julianhyde], do you want to chime in?
> Beeline doesn't accept --hiveconf option as Hive CLI does
> -
>
> Key: HIVE-6173
> URL: https://issues.apache.org/jira/browse/HIVE-6173
> Project: Hive
> Issue Type: Improvement
> Components: CLI
>Affects Versions: 0.10.0, 0.11.0, 0.12.0
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Labels: TODOC13
> Fix For: 0.13.0
>
> Attachments: HIVE-6173.1.patch, HIVE-6173.2.patch, HIVE-6173.patch
>
>
> {code}
> > beeline -u jdbc:hive2:// --hiveconf a=b
> Usage: java org.apache.hive.cli.beeline.BeeLine
> {code}
> Since Beeline is replacing Hive CLI, it should support this command line
> option as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9188) BloomFilter in ORC row group index
[ https://issues.apache.org/jira/browse/HIVE-9188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267032#comment-14267032 ] Prasanth Jayachandran commented on HIVE-9188: - This patch needs to be rebased after HIVE-4639 as both patches touches the same set of files. > BloomFilter in ORC row group index > -- > > Key: HIVE-9188 > URL: https://issues.apache.org/jira/browse/HIVE-9188 > Project: Hive > Issue Type: New Feature > Components: File Formats >Affects Versions: 0.15.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Labels: orcfile > Attachments: HIVE-9188.1.patch, HIVE-9188.2.patch, HIVE-9188.3.patch, > HIVE-9188.4.patch > > > BloomFilters are well known probabilistic data structure for set membership > checking. We can use bloom filters in ORC index for better row group pruning. > Currently, ORC row group index uses min/max statistics to eliminate row > groups (stripes as well) that do not satisfy predicate condition specified in > the query. But in some cases, the efficiency of min/max based elimination is > not optimal (unsorted columns with wide range of entries). Bloom filters can > be an effective and efficient alternative for row group/split elimination for > point queries or queries with IN clause. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-4639) Add has null flag to ORC internal index
[ https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-4639: --- Assignee: Prasanth Jayachandran (was: Owen O'Malley) > Add has null flag to ORC internal index > --- > > Key: HIVE-4639 > URL: https://issues.apache.org/jira/browse/HIVE-4639 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Owen O'Malley >Assignee: Prasanth Jayachandran > Attachments: HIVE-4639.1.patch > > > It would enable more predicate pushdown if we added a flag to the index entry > recording if there were any null values in the column for the 10k rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4639) Add has null flag to ORC internal index
[ https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-4639: Status: Patch Available (was: Open) > Add has null flag to ORC internal index > --- > > Key: HIVE-4639 > URL: https://issues.apache.org/jira/browse/HIVE-4639 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Owen O'Malley >Assignee: Prasanth Jayachandran > Attachments: HIVE-4639.1.patch > > > It would enable more predicate pushdown if we added a flag to the index entry > recording if there were any null values in the column for the 10k rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4639) Add has null flag to ORC internal index
[ https://issues.apache.org/jira/browse/HIVE-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-4639: Attachment: HIVE-4639.1.patch > Add has null flag to ORC internal index > --- > > Key: HIVE-4639 > URL: https://issues.apache.org/jira/browse/HIVE-4639 > Project: Hive > Issue Type: Improvement > Components: File Formats >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Attachments: HIVE-4639.1.patch > > > It would enable more predicate pushdown if we added a flag to the index entry > recording if there were any null values in the column for the 10k rows. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Building Hive-0.14 is failing because artifact pentaho-aggdesigner-algorithm-5.1.3-jhyde could not be resolved
Hi folks! I'm still stuck on this issue. Hopefully people are back from their vacations and can now reply. Resending. On Monday, December 29, 2014 1:17 PM, Ravi Prakash wrote: Hi folks! I'm still somewhat stuck on this issue. Could someone please guide me? I had to manually download these jars and put them into my ~/.m2/repository from http://repo.spring.io/libs-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.3-jhyde/ . (When I configured this repository, maven somehow still didn't find the artifacts) I had to do this for the following artifacts:http://repo.spring.io/libs-release/org/pentaho/pentaho-aggdesigner/5.1.3-jhyde/pentaho-aggdesigner-5.1.3-jhyde.pomhttp://repo.spring.io/libs-release/net/hydromatic/linq4j/0.4/linq4j-0.4.jarhttp://repo.spring.io/libs-release/net/hydromatic/quidem/0.1.1/quidem-0.1.1.pom Consequently I was able to build hive, however I don't believe this is a kosher way of producing production binaries, so I am still waiting to hear from you all. Is there another trusted repository I can try which has these artifacts indexed? ThanksRavi On Monday, December 22, 2014 2:14 PM, Ravi Prakash wrote: Hi! Has anyone tried building Hive-0.14 from source? I'm using the tag for release-0.14.0 https://github.com/apache/hive/releases/tag/release-0.14.0 The command I use is: mvn install -DskipTests -Phadoop-2 -DcreateChecksum=true -Dtez.version=0.5.3 -Dcalcite.version=0.9.2-incubating The build fails for me with the following error:[ERROR] Failed to execute goal on project hive-exec: Could not resolve dependencies for project org.apache.hive:hive-exec:jar:0.14.0: The following artifacts could not be resolved: org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde, net.hydromatic:linq4j:jar:0.4, net.hydromatic:quidem:jar:0.1.1: Could not find artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde in nexus (http://localhost:8081/nexus/content/groups/public) -> [Help 1] This is a transitive dependency via the "calcite-0.9.2-incubating" artifact. Is there a JIRA which someone can please point me to? It seems wrong that an artifact with version "5.1.3-jhyde" is required to build Apache Hive, no disrespect to Julian. Am I missing something? ThanksRavi
[jira] [Commented] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266990#comment-14266990 ] Vaibhav Gumashta commented on HIVE-6679: Committed to branch 14. [~thejas] I'll create a new patch for trunk with just the socket timeout code. > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0 >Reporter: Prasad Mujumdar >Assignee: Navis > Fix For: 0.15.0, 0.14.1 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch > > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-6679) HiveServer2 should support configurable the server side socket timeout and keepalive for various transports types where applicable
[ https://issues.apache.org/jira/browse/HIVE-6679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-6679: --- Fix Version/s: 0.15.0 > HiveServer2 should support configurable the server side socket timeout and > keepalive for various transports types where applicable > -- > > Key: HIVE-6679 > URL: https://issues.apache.org/jira/browse/HIVE-6679 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.13.0, 0.14.0 >Reporter: Prasad Mujumdar >Assignee: Navis > Fix For: 0.15.0, 0.14.1 > > Attachments: HIVE-6679.1.patch.txt, HIVE-6679.2.patch.txt, > HIVE-6679.3.patch, HIVE-6679.4.patch, HIVE-6679.5.patch > > > HiveServer2 should support configurable the server side socket read timeout > and TCP keep-alive option. Metastore server already support this (and the so > is the old hive server). > We now have multiple client connectivity options like Kerberos, Delegation > Token (Digest-MD5), Plain SASL, Plain SASL with SSL and raw sockets. The > configuration should be applicable to all types (if possible). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9267) Ensure custom UDF works with Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266987#comment-14266987
]
Hive QA commented on HIVE-9267:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690425/HIVE-9267.1-spark.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 7285 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_example_add
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/610/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/610/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-610/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690425 - PreCommit-HIVE-SPARK-Build
> Ensure custom UDF works with Spark [Spark Branch]
> -
>
> Key: HIVE-9267
> URL: https://issues.apache.org/jira/browse/HIVE-9267
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Xuefu Zhang
>Assignee: Xuefu Zhang
> Attachments: HIVE-9267.1-spark.patch
>
>
> Create or add auto qtest if necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9183) Compilation against HBase 1.0.0 fails due to missing dependency on high_scale_lib.Counter
[
https://issues.apache.org/jira/browse/HIVE-9183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-9183:
---
Resolution: Fixed
Fix Version/s: 0.15.0
Status: Resolved (was: Patch Available)
committed to trunk. Thanks Ted!
> Compilation against HBase 1.0.0 fails due to missing dependency on
> high_scale_lib.Counter
> -
>
> Key: HIVE-9183
> URL: https://issues.apache.org/jira/browse/HIVE-9183
> Project: Hive
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.15.0
>
> Attachments: HIVE-9183-001.patch, HIVE-9183.patch, HIVE-9183.patch
>
>
> In the HBase 1.0.0 release, org.cliffc.high_scale_lib is no longer used.
> HBaseStorageHandler.java has compile time dependency on high_scale_lib.Counter
> Compilation against HBase 1.0.0 release would fail due to the missing
> dependency.
> The following change would allow compilation to pass:
> {code}
> diff --git
> a/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
>
> b/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
> index 3218639..6f626f0 100644
> ---
> a/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
> +++
> b/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
> @@ -499,8 +499,8 @@ public void configureJobConf(TableDesc tableDesc, JobConf
> jobConf) {
> * only need TableMapReduceUtil.addDependencyJars(jobConf) here.
> */
>TableMapReduceUtil.addDependencyJars(
> - jobConf, HBaseStorageHandler.class, TableInputFormatBase.class,
> - org.cliffc.high_scale_lib.Counter.class); // this will be removed
> for HBase 1.0
> + jobConf, HBaseStorageHandler.class, TableInputFormatBase.class);
> + // org.cliffc.high_scale_lib.Counter.class); this will be removed
> for HBase 1.0
>Set merged = new
> LinkedHashSet(jobConf.getStringCollection("tmpjars"));
>Job copy = new Job(jobConf);
> {code}
> However, loading of high_scale_lib.Counter should be done at runtime so that
> 0.98 and 1.0 releases are both supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9273) Add option to fire metastore event on insert
Alan Gates created HIVE-9273: Summary: Add option to fire metastore event on insert Key: HIVE-9273 URL: https://issues.apache.org/jira/browse/HIVE-9273 Project: Hive Issue Type: New Feature Reporter: Alan Gates Assignee: Alan Gates HIVE-9271 adds the ability for the client to request firing metastore events. This can be used in the MoveTask to fire events when an insert is done that does not add partitions to a table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
