[jira] [Updated] (HIVE-14735) Build Infra: Spark artifacts download takes a long time
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-14735: Description: In particular this command: {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} was: In particular this command in build.sh: {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} > Build Infra: Spark artifacts download takes a long time > --- > > Key: HIVE-14735 > URL: https://issues.apache.org/jira/browse/HIVE-14735 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Vaibhav Gumashta > > In particular this command: > {{curl -Sso ./../thirdparty/spark-1.6.0-bin-hadoop2-without-hive.tgz > http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.6.0-bin-hadoop2-without-hive.tgz}} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13878: Status: Patch Available (was: In Progress) Avoid rescanning string/char/varchar to determine if there were escapes. > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, > HIVE-13878.09.patch, HIVE-13878.1.patch, HIVE-13878.2.patch, > HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13878: Attachment: HIVE-13878.09.patch > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, > HIVE-13878.09.patch, HIVE-13878.1.patch, HIVE-13878.2.patch, > HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13878: Status: In Progress (was: Patch Available) > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, > HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12812) Enable mapred.input.dir.recursive by default to support union with aggregate function
[ https://issues.apache.org/jira/browse/HIVE-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483002#comment-15483002 ] Hive QA commented on HIVE-12812: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12827953/HIVE-12812.patch {color:green}SUCCESS:{color} +1 due to 54 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10546 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[root_dir_external_table] org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[list_bucket_dml_2] org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testTaskStatus {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1151/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1151/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1151/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 11 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12827953 - PreCommit-HIVE-MASTER-Build > Enable mapred.input.dir.recursive by default to support union with aggregate > function > - > > Key: HIVE-12812 > URL: https://issues.apache.org/jira/browse/HIVE-12812 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1, 2.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12812.patch, HIVE-12812.patch > > > When union remove optimization is enabled, union query with aggregate > function writes its subquery intermediate results to subdirs which needs > mapred.input.dir.recursive to be enabled in order to be fetched. This > property is not defined by default in Hive and often ignored by user, which > causes the query failure and is hard to be debugged. > So we need set mapred.input.dir.recursive to true whenever union remove > optimization is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482932#comment-15482932 ] Matt McCline commented on HIVE-13878: - Test failures are not related. > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, > HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482896#comment-15482896 ] Rui Li commented on HIVE-14412: --- The failure don't seem related. [~xuefuz] could you take a look? I've put the v4 patch to RB. Thanks. > Add a timezone-aware timestamp > -- > > Key: HIVE-14412 > URL: https://issues.apache.org/jira/browse/HIVE-14412 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Rui Li >Assignee: Rui Li > Attachments: HIVE-14412.1.patch, HIVE-14412.2.patch, > HIVE-14412.3.patch, HIVE-14412.4.patch > > > Java's Timestamp stores the time elapsed since the epoch. While it's by > itself unambiguous, ambiguity comes when we parse a string into timestamp, or > convert a timestamp to string, causing problems like HIVE-14305. > To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13589) beeline - support prompt for password with '-u' option
[ https://issues.apache.org/jira/browse/HIVE-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482879#comment-15482879 ] Ke Jia commented on HIVE-13589: --- Hi [~vihangk1], Thanks for your comments. About this issue, I have some ideas and now am working on the patch. May I continue my work for this JIRA? The main idea is that we can add the "-- force" or some other "--" options in Beeline.java, which are defined in https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineCommands. You can share your thoughts once my patch is ready. Thanks again for your reviews and inputs. > beeline - support prompt for password with '-u' option > -- > > Key: HIVE-13589 > URL: https://issues.apache.org/jira/browse/HIVE-13589 > Project: Hive > Issue Type: Bug > Components: Beeline >Reporter: Thejas M Nair >Assignee: Ke Jia > Fix For: 2.2.0 > > Attachments: HIVE-13589.1.patch, HIVE-13589.2.patch, > HIVE-13589.3.patch, HIVE-13589.4.patch, HIVE-13589.5.patch, HIVE-13589.6.patch > > > Specifying connection string using commandline options in beeline is > convenient, as it gets saved in shell command history, and it is easy to > retrieve it from there. > However, specifying the password in command prompt is not secure as it gets > displayed on screen and saved in the history. > It should be possible to specify '-p' without an argument to make beeline > prompt for password. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12812) Enable mapred.input.dir.recursive by default to support union with aggregate function
[ https://issues.apache.org/jira/browse/HIVE-12812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoyu Tang updated HIVE-12812: --- Attachment: HIVE-12812.patch The failed tests seem not related to this patch. Reattach it to kick off another precommit test. > Enable mapred.input.dir.recursive by default to support union with aggregate > function > - > > Key: HIVE-12812 > URL: https://issues.apache.org/jira/browse/HIVE-12812 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1, 2.1.0 >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-12812.patch, HIVE-12812.patch > > > When union remove optimization is enabled, union query with aggregate > function writes its subquery intermediate results to subdirs which needs > mapred.input.dir.recursive to be enabled in order to be fetched. This > property is not defined by default in Hive and often ignored by user, which > causes the query failure and is hard to be debugged. > So we need set mapred.input.dir.recursive to true whenever union remove > optimization is enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically
[ https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482800#comment-15482800 ] Sergio Peña commented on HIVE-14734: Btw, the idea is to call {{jenkins-execute-build.sh}} from the {{PreCommit-HIVE-Build}} job. This job should run tests for any branch, and the branch name should appear on the build jenkins list. For instance: {noformat} 150 Sep 12, 2016 8:18 PM HIVE-13878 branch-2.1 149 Sep 11, 2016 8:18 PM HIVE-13878 master {noformat} > Allow jenkins ptest job to execute tests on branch dynamically > -- > > Key: HIVE-14734 > URL: https://issues.apache.org/jira/browse/HIVE-14734 > Project: Hive > Issue Type: Task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14734 > > > NO PRECOMMIT TESTS > Currently, to execute tests on a new branch, a manual process must be done: > 1. Create a new Jenkins job with the new branch name > 2. Create a patch to jenkins-submit-build.sh with the new branch > 3. Create a profile properties file on the ptest master with the new branch > This jira will attempt to automate steps 1 and 2 by detecting the branch > profile from a patch to test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482797#comment-15482797 ] Hive QA commented on HIVE-13878: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12827941/HIVE-13878.08.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10546 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1150/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1150/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1150/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12827941 - PreCommit-HIVE-MASTER-Build > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, > HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically
[ https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14734: --- Status: Patch Available (was: Open) > Allow jenkins ptest job to execute tests on branch dynamically > -- > > Key: HIVE-14734 > URL: https://issues.apache.org/jira/browse/HIVE-14734 > Project: Hive > Issue Type: Task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14734 > > > NO PRECOMMIT TESTS > Currently, to execute tests on a new branch, a manual process must be done: > 1. Create a new Jenkins job with the new branch name > 2. Create a patch to jenkins-submit-build.sh with the new branch > 3. Create a profile properties file on the ptest master with the new branch > This jira will attempt to automate steps 1 and 2 by detecting the branch > profile from a patch to test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically
[ https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482790#comment-15482790 ] Sergio Peña commented on HIVE-14734: [~sseth] [~prasanth_j] Can you help me review this patch? The patch will detect the branch profile from the file attached to the JIRA, and then will call ptest with such profile. I also made some refactoring on the 'jenkins-execute-build.sh' script. > Allow jenkins ptest job to execute tests on branch dynamically > -- > > Key: HIVE-14734 > URL: https://issues.apache.org/jira/browse/HIVE-14734 > Project: Hive > Issue Type: Task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14734 > > > NO PRECOMMIT TESTS > Currently, to execute tests on a new branch, a manual process must be done: > 1. Create a new Jenkins job with the new branch name > 2. Create a patch to jenkins-submit-build.sh with the new branch > 3. Create a profile properties file on the ptest master with the new branch > This jira will attempt to automate steps 1 and 2 by detecting the branch > profile from a patch to test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically
[ https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14734: --- Description: NO PRECOMMIT TESTS Currently, to execute tests on a new branch, a manual process must be done: 1. Create a new Jenkins job with the new branch name 2. Create a patch to jenkins-submit-build.sh with the new branch 3. Create a profile properties file on the ptest master with the new branch This jira will attempt to automate steps 1 and 2 by detecting the branch profile from a patch to test. was: Currently, to execute tests on a new branch, a manual process must be done: 1. Create a new Jenkins job with the new branch name 2. Create a patch to jenkins-submit-build.sh with the new branch 3. Create a profile properties file on the ptest master with the new branch This jira will attempt to automate steps 1 and 2 by detecting the branch profile from a patch to test. > Allow jenkins ptest job to execute tests on branch dynamically > -- > > Key: HIVE-14734 > URL: https://issues.apache.org/jira/browse/HIVE-14734 > Project: Hive > Issue Type: Task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14734 > > > NO PRECOMMIT TESTS > Currently, to execute tests on a new branch, a manual process must be done: > 1. Create a new Jenkins job with the new branch name > 2. Create a patch to jenkins-submit-build.sh with the new branch > 3. Create a profile properties file on the ptest master with the new branch > This jira will attempt to automate steps 1 and 2 by detecting the branch > profile from a patch to test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically
[ https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14734: --- Attachment: HIVE-14734 > Allow jenkins ptest job to execute tests on branch dynamically > -- > > Key: HIVE-14734 > URL: https://issues.apache.org/jira/browse/HIVE-14734 > Project: Hive > Issue Type: Task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14734 > > > NO PRECOMMIT TESTS > Currently, to execute tests on a new branch, a manual process must be done: > 1. Create a new Jenkins job with the new branch name > 2. Create a patch to jenkins-submit-build.sh with the new branch > 3. Create a profile properties file on the ptest master with the new branch > This jira will attempt to automate steps 1 and 2 by detecting the branch > profile from a patch to test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14734) Allow jenkins ptest job to execute tests on branch dynamically
[ https://issues.apache.org/jira/browse/HIVE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14734: --- Description: Currently, to execute tests on a new branch, a manual process must be done: 1. Create a new Jenkins job with the new branch name 2. Create a patch to jenkins-submit-build.sh with the new branch 3. Create a profile properties file on the ptest master with the new branch This jira will attempt to automate steps 1 and 2 by detecting the branch profile from a patch to test. was: Currently, to execute tests on a new branch, a manual process must be done to: 1. Create a new Jenkins job with the new branch name 2. Create a patch to jenkins-submit-build.sh with the new branch 3. Create a profile properties file on the ptest master with the new branch This jira will attempt to automate steps 1 and 2 for any branch that is submitted for testing. > Allow jenkins ptest job to execute tests on branch dynamically > -- > > Key: HIVE-14734 > URL: https://issues.apache.org/jira/browse/HIVE-14734 > Project: Hive > Issue Type: Task > Components: Hive, Testing Infrastructure >Reporter: Sergio Peña >Assignee: Sergio Peña > > Currently, to execute tests on a new branch, a manual process must be done: > 1. Create a new Jenkins job with the new branch name > 2. Create a patch to jenkins-submit-build.sh with the new branch > 3. Create a profile properties file on the ptest master with the new branch > This jira will attempt to automate steps 1 and 2 by detecting the branch > profile from a patch to test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14626) Support Trash in Truncate Table
[ https://issues.apache.org/jira/browse/HIVE-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482740#comment-15482740 ] Chaoyu Tang commented on HIVE-14626: Thanks, [~leftylev]. The patch is to enhance the Truncate with Trash support and there is not backward compatibility, so I do not think we need the explanation to the behavior before this. > Support Trash in Truncate Table > --- > > Key: HIVE-14626 > URL: https://issues.apache.org/jira/browse/HIVE-14626 > Project: Hive > Issue Type: Sub-task > Components: Query Processor >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14626.1.patch, HIVE-14626.patch > > > Currently Truncate Table (or Partition) is implemented using > FileSystem.delete and then recreate the directory, so > 1. it does not support HDFS Trash > 2. if the table/partition directory is initially encryption protected, after > being deleted and recreated, it is no more protected. > The new implementation is to clean the contents of directory using > multi-threaded trashFiles. If Trash is enabled and has a lower encryption > level than the data directory, the files under it will be deleted. Otherwise, > they will be Trashed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14727) llap-server may case file descriptor leak in BuddyAllocator class
[ https://issues.apache.org/jira/browse/HIVE-14727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482720#comment-15482720 ] Yechao Chen commented on HIVE-14727: There is no original failrue,I just read the llap-server code ,find this may case this suitation > llap-server may case file descriptor leak in BuddyAllocator class > - > > Key: HIVE-14727 > URL: https://issues.apache.org/jira/browse/HIVE-14727 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Yechao Chen >Assignee: Yechao Chen > Fix For: 2.2.0 > > Attachments: HIVE-14727.patch > > > llap-server,the method preallocate(int) of BuddyAllocator may case file > descriptor leak when FileChannel map allocate memory error. > the code: > //here if failed > ByteBuffer rwbuf = rwf.getChannel().map(MapMode.READ_WRITE, 0, > arenaSize); > // A mapping, once established, is not dependent upon the file > channel that was used to > // create it. delete file and hold onto the map >//can not close() and delete file > rwf.close(); > rf.delete(); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13878: Status: Patch Available (was: In Progress) > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, > HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13878: Status: In Progress (was: Patch Available) > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, > HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13878: Attachment: HIVE-13878.08.patch > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.08.patch, > HIVE-13878.1.patch, HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482557#comment-15482557 ] Matt McCline commented on HIVE-13878: - Test failure TestVectorSerDeRow *is* related. > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.1.patch, > HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15482456#comment-15482456 ] Hive QA commented on HIVE-13878: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12827936/HIVE-13878.07.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10546 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] org.apache.hadoop.hive.ql.exec.vector.TestVectorSerDeRow.testVectorSerDeRow org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1149/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1149/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1149/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12827936 - PreCommit-HIVE-MASTER-Build > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.1.patch, > HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13878: Status: Patch Available (was: In Progress) Optimize LazySimpleDeserializeRead.parse > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.1.patch, > HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13878: Status: In Progress (was: Patch Available) > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.1.patch, > HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13878: Attachment: HIVE-13878.07.patch > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.07.patch, HIVE-13878.1.patch, > HIVE-13878.2.patch, HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14686) Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS ... AS"
[ https://issues.apache.org/jira/browse/HIVE-14686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481681#comment-15481681 ] Fan Yunbo commented on HIVE-14686: -- thanks for you review. > Get unexpected command type when execute query "CREATE TABLE IF NOT EXISTS > ... AS" > -- > > Key: HIVE-14686 > URL: https://issues.apache.org/jira/browse/HIVE-14686 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.1, 2.1.0 >Reporter: Fan Yunbo >Assignee: Fan Yunbo > Fix For: 2.2.0 > > Attachments: HIVE-14686.1.patch, HIVE-14686.2.patch, > HIVE-14686.3.patch, HIVE-14686.4.patch, HIVE-14686.5.patch > > > See the query: > {code} > create table if not exists DST as select * from SRC; > {code} > if the table DST doesn't exist, SessionState.get().getHiveOperation() will > return HiveOperation.CREATETABLE_AS_SELECT; > But if the table DST already exists, it will return HiveOperation.CREATETABLE; > It really makes some trouble for those who judge operation type by > SessionState.get().getHiveOperation(). > The reason I find out is that the function analyzeCreateTable in > SemanticAnalyzer.java will return null and won't set the correct command type > if the table already exists. > Here is the related code: > {code} > // check for existence of table > if (ifNotExists) { > try { > Table table = getTable(qualifiedTabName, false); > if (table != null) { // table exists > return null; > } > } catch (HiveException e) { > // should not occur since second parameter to getTableWithQN is false > throw new IllegalStateException("Unxpected Exception thrown: " + > e.getMessage(), e); > } > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481172#comment-15481172 ] Gopal V commented on HIVE-13878: Thanks [~mmccline], I've added this to my build queue. > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.1.patch, HIVE-13878.2.patch, > HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481169#comment-15481169 ] Matt McCline commented on HIVE-13878: - Test failures are unrelated. > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.1.patch, HIVE-13878.2.patch, > HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13878) Vectorization: Column pruning for Text vectorization
[ https://issues.apache.org/jira/browse/HIVE-13878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481161#comment-15481161 ] Hive QA commented on HIVE-13878: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12827916/HIVE-13878.06.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10544 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats0] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char] org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning] org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1148/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1148/console Test logs: http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1148/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12827916 - PreCommit-HIVE-MASTER-Build > Vectorization: Column pruning for Text vectorization > > > Key: HIVE-13878 > URL: https://issues.apache.org/jira/browse/HIVE-13878 > Project: Hive > Issue Type: Bug > Components: Vectorization >Affects Versions: 2.1.0 >Reporter: Gopal V >Assignee: Matt McCline > Attachments: HIVE-13878.04.patch, HIVE-13878.05.patch, > HIVE-13878.06.patch, HIVE-13878.1.patch, HIVE-13878.2.patch, > HIVE-13878.3.patch > > > Column pruning in TextFile vectorization does not work with Vector SerDe > settings due to LazySimple deser codepath issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14626) Support Trash in Truncate Table
[ https://issues.apache.org/jira/browse/HIVE-14626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481159#comment-15481159 ] Lefty Leverenz commented on HIVE-14626: --- Thanks for the doc and the jira for encryption, [~ctang.ma]. I had forgotten about HIVE-8065, which still needs to be documented in the wiki. I added version information in the Truncate Table section, with a link to this issue. Should we also explain what the behavior was before this patch? > Support Trash in Truncate Table > --- > > Key: HIVE-14626 > URL: https://issues.apache.org/jira/browse/HIVE-14626 > Project: Hive > Issue Type: Sub-task > Components: Query Processor >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14626.1.patch, HIVE-14626.patch > > > Currently Truncate Table (or Partition) is implemented using > FileSystem.delete and then recreate the directory, so > 1. it does not support HDFS Trash > 2. if the table/partition directory is initially encryption protected, after > being deleted and recreated, it is no more protected. > The new implementation is to clean the contents of directory using > multi-threaded trashFiles. If Trash is enabled and has a lower encryption > level than the data directory, the files under it will be deleted. Otherwise, > they will be Trashed -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8065) Support HDFS encryption functionality on Hive
[ https://issues.apache.org/jira/browse/HIVE-8065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15481156#comment-15481156 ] Lefty Leverenz commented on HIVE-8065: -- Nudge: [~spena], could you please document HDFS encryption in the wiki? Or if you don't have time, could you suggest someone else to do it? Thanks. > Support HDFS encryption functionality on Hive > - > > Key: HIVE-8065 > URL: https://issues.apache.org/jira/browse/HIVE-8065 > Project: Hive > Issue Type: Improvement >Affects Versions: 0.13.1 >Reporter: Sergio Peña >Assignee: Sergio Peña > Labels: TODOC15 > > The new encryption support on HDFS makes Hive incompatible and unusable when > this feature is used. > HDFS encryption is designed so that an user can configure different > encryption zones (or directories) for multi-tenant environments. An > encryption zone has an exclusive encryption key, such as AES-128 or AES-256. > Because of security compliance, the HDFS does not allow to move/rename files > between encryption zones. Renames are allowed only inside the same encryption > zone. A copy is allowed between encryption zones. > See HDFS-6134 for more details about HDFS encryption design. > Hive currently uses a scratch directory (like /tmp/$user/$random). This > scratch directory is used for the output of intermediate data (between MR > jobs) and for the final output of the hive query which is later moved to the > table directory location. > If Hive tables are in different encryption zones than the scratch directory, > then Hive won't be able to renames those files/directories, and it will make > Hive unusable. > To handle this problem, we can change the scratch directory of the > query/statement to be inside the same encryption zone of the table directory > location. This way, the renaming process will be successful. > Also, for statements that move files between encryption zones (i.e. LOAD > DATA), a copy may be executed instead of a rename. This will cause an > overhead when copying large data files, but it won't break the encryption on > Hive. > Another security thing to consider is when using joins selects. If Hive joins > different tables with different encryption key strengths, then the results of > the select might break the security compliance of the tables. Let's say two > tables with 128 bits and 256 bits encryption are joined, then the temporary > results might be stored in the 128 bits encryption zone. This will conflict > with the table encrypted with 256 bits temporary. > To fix this, Hive should be able to select the scratch directory that is more > secured/encrypted in order to save the intermediate data temporary with no > compliance issues. > For instance: > {noformat} > SELECT * FROM table-aes128 t1 JOIN table-aes256 t2 WHERE t1.id == t2.id; > {noformat} > - This should use a scratch directory (or staging directory) inside the > table-aes256 table location. > {noformat} > INSERT OVERWRITE TABLE table-unencrypted SELECT * FROM table-aes1; > {noformat} > - This should use a scratch directory inside the table-aes1 location. > {noformat} > FROM table-unencrypted > INSERT OVERWRITE TABLE table-aes128 SELECT id, name > INSERT OVERWRITE TABLE table-aes256 SELECT id, name > {noformat} > - This should use a scratch directory on each of the tables locations. > - The first SELECT will have its scratch directory on table-aes128 directory. > - The second SELECT will have its scratch directory on table-aes256 directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)