[jira] [Created] (HIVE-10666) Improvement for Parquet Predicate Push down on Hive
Dong Chen created HIVE-10666: Summary: Improvement for Parquet Predicate Push down on Hive Key: HIVE-10666 URL: https://issues.apache.org/jira/browse/HIVE-10666 Project: Hive Issue Type: Improvement Reporter: Dong Chen Assignee: Dong Chen This umbrella JIRA will contain the improvements for Parquet PPD on Hive, which fall into: 1. date types support 2. bug fix 3. performance -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10642) Update Timestamp type mapping to Parquet
Dong Chen created HIVE-10642: Summary: Update Timestamp type mapping to Parquet Key: HIVE-10642 URL: https://issues.apache.org/jira/browse/HIVE-10642 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Timestamp in Hive was originally mapped to INT_96 in Parquet before. And inside Parquet it use Binary to save values. When working on HIVE-10255, I found predicate with timestamp does not work, since comparing Binary representative of Timestamp value can not give correct relationship. Meanwhile, in latest version Parquet, the timestamp is mapped to INT_64. Refer to https://github.com/Parquet/parquet-format/blob/master/LogicalTypes.md Therefore, this Jira will update the mapping of Timestamp type in Hive, and keep it backward compatible. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10257) Ensure Parquet Hive has null optimization
Dong Chen created HIVE-10257:
Summary: Ensure Parquet Hive has null optimization
Key: HIVE-10257
URL: https://issues.apache.org/jira/browse/HIVE-10257
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
In Parquet statistics, a boolean value {{hasNonNullValue}} is used for each
column chunk. Hive could use this value to skip a column, avoid null-checking
logic, and speed up vectorization like HIVE-4478 (in the future, it is not
completed yet).
In this Jira we could check whether this null optimization works, and make
changes if any.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10256) Eliminate row groups based on the block statistics in Parquet
Dong Chen created HIVE-10256:
Summary: Eliminate row groups based on the block statistics in
Parquet
Key: HIVE-10256
URL: https://issues.apache.org/jira/browse/HIVE-10256
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
In Parquet PPD, the not matched row groups should be eliminated. See
{{TestOrcSplitElimination}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10255) Parquet PPD support TIMESTAMP
Dong Chen created HIVE-10255: Summary: Parquet PPD support TIMESTAMP Key: HIVE-10255 URL: https://issues.apache.org/jira/browse/HIVE-10255 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10253) Parquet PPD support DATE
Dong Chen created HIVE-10253: Summary: Parquet PPD support DATE Key: HIVE-10253 URL: https://issues.apache.org/jira/browse/HIVE-10253 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Hive should handle the DATE data type when generating and pushing the predicate to Parquet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10254) Parquet PPD support DECIMAL
Dong Chen created HIVE-10254: Summary: Parquet PPD support DECIMAL Key: HIVE-10254 URL: https://issues.apache.org/jira/browse/HIVE-10254 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10252) Make PPD work for Parquet in row group level
Dong Chen created HIVE-10252:
Summary: Make PPD work for Parquet in row group level
Key: HIVE-10252
URL: https://issues.apache.org/jira/browse/HIVE-10252
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
In Hive, predicate pushdown figures out the search condition in HQL, serialize
it, and push to file format. ORC could use the predicate to filter stripes.
Similarly, Parquet should use the statics saved in row group to filter not
match row group. But it does not work.
In {{ParquetRecordReaderWrapper}}, it get splits with all row groups (client
side), and push the filter to Parquet for further processing (parquet side).
But in {{ParquetRecordReader.initializeInternalReader()}}, if the splits have
already been selected by client side, it will not handle filter again.
We should make the behavior consistent in Hive. Maybe we could get splits,
filter them, and then pass to parquet. This means using client side strategy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10031) Modify the using of jobConf variable in ParquetRecordReaderWrapper constructor
Dong Chen created HIVE-10031:
Summary: Modify the using of jobConf variable in
ParquetRecordReaderWrapper constructor
Key: HIVE-10031
URL: https://issues.apache.org/jira/browse/HIVE-10031
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
In {{ParquetRecordReaderWrapper}} constructor, it create splits, set
projections and filters in conf, create task context, and then create Parquet
record reader. In this procedure, we could improve the logic of conf usage:
1. the clone of jobConf is not necessary. This could speed up getRecordReader a
little.
2. the updated jobConf is not passed to Parquet in one case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10016) Remove duplicated Hive table schema parsing in DataWritableReadSupport
Dong Chen created HIVE-10016:
Summary: Remove duplicated Hive table schema parsing in
DataWritableReadSupport
Key: HIVE-10016
URL: https://issues.apache.org/jira/browse/HIVE-10016
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
In {{DataWritableReadSupport.init()}}, the table schema is created and its
string format is set in conf. When construct the {{ParquetRecordReaderWrapper}}
, the schema is fetched from conf and parsed several times.
We could remove these schema parsing, and improve the speed of getRecordReader
a bit.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9906) Add timeout mechanism in RawStoreProxy
Dong Chen created HIVE-9906: --- Summary: Add timeout mechanism in RawStoreProxy Key: HIVE-9906 URL: https://issues.apache.org/jira/browse/HIVE-9906 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen In HIVE-9253, we add a timeout mechanism in HMS. We start the timer in RetryingHMSHandler.invoke, and then -> RawStoreProxy.invoke -> ObjectStore.xxxMethod. The timer is stopped after methods complete. It was found that, the methods of ObjectStore might be invoked directly in o.a.h.h.ql.txn.compactor.CompactorThread, but not throught HMSHandler. This will cause timeout checking to throw exception. We need fix this bug here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9894) Use new parquet Types API builder to construct DATE data type
Dong Chen created HIVE-9894: --- Summary: Use new parquet Types API builder to construct DATE data type Key: HIVE-9894 URL: https://issues.apache.org/jira/browse/HIVE-9894 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen The DATE type was implemented in HIVE-8119. And new parquet Types API builder was used in HIVE-9657 for all data types. But DATE is missed. We should also use new Types API for DATE type. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9868) Turn on Parquet vectorization in parquet branch
Dong Chen created HIVE-9868: --- Summary: Turn on Parquet vectorization in parquet branch Key: HIVE-9868 URL: https://issues.apache.org/jira/browse/HIVE-9868 Project: Hive Issue Type: Sub-task Affects Versions: parquet-branch Reporter: Dong Chen Assignee: Dong Chen Parquet vectorization was turned off in HIVE-9235 due to data types issue. As the vectorization refactor work is starting in HIVE-8128 on parquet branch, let's turn on it on branch at first. The data types will be handled in refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9819) Add timeout check inside the HMS server
Dong Chen created HIVE-9819: --- Summary: Add timeout check inside the HMS server Key: HIVE-9819 URL: https://issues.apache.org/jira/browse/HIVE-9819 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen In HIVE-9253, a timeout check mechanism is added for long running methods in HMS server. We should add this check to each of the inner loops inside the HMS server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8128) Improve Parquet Vectorization
[ https://issues.apache.org/jira/browse/HIVE-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319723#comment-14319723 ] Dong Chen commented on HIVE-8128: - Will start from a POC based on the new vectorized Parquet API at https://github.com/zhenxiao/incubator-parquet-mr/pull/1 > Improve Parquet Vectorization > - > > Key: HIVE-8128 > URL: https://issues.apache.org/jira/browse/HIVE-8128 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Dong Chen > > We'll want to do is finish the vectorization work (e.g. VectorizedOrcSerde, > VectorizedOrcSerde) which was partially done in HIVE-5998. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-8128) Improve Parquet Vectorization
[ https://issues.apache.org/jira/browse/HIVE-8128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-8128 started by Dong Chen. --- > Improve Parquet Vectorization > - > > Key: HIVE-8128 > URL: https://issues.apache.org/jira/browse/HIVE-8128 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Dong Chen > > We'll want to do is finish the vectorization work (e.g. VectorizedOrcSerde, > VectorizedOrcSerde) which was partially done in HIVE-5998. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9332) Warn user when parquet mm kicks in
[ https://issues.apache.org/jira/browse/HIVE-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319683#comment-14319683 ] Dong Chen commented on HIVE-9332: - PARQUET-164 is on-going. Will start this after warning mechanism is done in Parquet side. > Warn user when parquet mm kicks in > -- > > Key: HIVE-9332 > URL: https://issues.apache.org/jira/browse/HIVE-9332 > Project: Hive > Issue Type: Sub-task > Components: File Formats >Reporter: Brock Noland >Assignee: Dong Chen > > In PARQUET-164 will add a warning mechanism when the mm kicks in. It'd be > great if we could feed that warning back to the user in form of a console > warning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-9332) Warn user when parquet mm kicks in
[ https://issues.apache.org/jira/browse/HIVE-9332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen reassigned HIVE-9332: --- Assignee: Dong Chen > Warn user when parquet mm kicks in > -- > > Key: HIVE-9332 > URL: https://issues.apache.org/jira/browse/HIVE-9332 > Project: Hive > Issue Type: Sub-task > Components: File Formats >Reporter: Brock Noland >Assignee: Dong Chen > > In PARQUET-164 will add a warning mechanism when the mm kicks in. It'd be > great if we could feed that warning back to the user in form of a console > warning. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8119) Implement Date in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-8119: Attachment: HIVE-8119.1.patch Patch v1. A slight change to import class DateWritable. > Implement Date in ParquetSerde > -- > > Key: HIVE-8119 > URL: https://issues.apache.org/jira/browse/HIVE-8119 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Mohit Sabharwal > Attachments: HIVE-8119.1.patch, HIVE-8119.patch > > > Date type in Parquet is discussed here: > http://mail-archives.apache.org/mod_mbox/incubator-parquet-dev/201406.mbox/%3CCAKa9qDkp7xn+H8fNZC7ms3ckd=xr8gdpe7gqgj5o+pybdem...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8119) Implement Date in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308800#comment-14308800 ] Dong Chen commented on HIVE-8119: - RB: https://reviews.apache.org/r/30717/ > Implement Date in ParquetSerde > -- > > Key: HIVE-8119 > URL: https://issues.apache.org/jira/browse/HIVE-8119 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Mohit Sabharwal > Attachments: HIVE-8119.patch > > > Date type in Parquet is discussed here: > http://mail-archives.apache.org/mod_mbox/incubator-parquet-dev/201406.mbox/%3CCAKa9qDkp7xn+H8fNZC7ms3ckd=xr8gdpe7gqgj5o+pybdem...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8119) Implement Date in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-8119: Attachment: HIVE-8119.patch Hi [~mohitsabharwal], I am working on HIVE-9442 to check all data types for PARQUET, and reusing some existed test cases needs DATE type working. I got a patch to implement DATE in PARQUET, could you please review it if time is available? Sorry for interrupting your work on it. :) Based on Parquet Logical Type Definitions in https://github.com/apache/incubator-parquet-format/blob/master/LogicalTypes.md, this patch map the Date in Hive to INT32 in Parquet. > Implement Date in ParquetSerde > -- > > Key: HIVE-8119 > URL: https://issues.apache.org/jira/browse/HIVE-8119 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Mohit Sabharwal > Attachments: HIVE-8119.patch > > > Date type in Parquet is discussed here: > http://mail-archives.apache.org/mod_mbox/incubator-parquet-dev/201406.mbox/%3CCAKa9qDkp7xn+H8fNZC7ms3ckd=xr8gdpe7gqgj5o+pybdem...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8119) Implement Date in ParquetSerde
[ https://issues.apache.org/jira/browse/HIVE-8119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-8119: Status: Patch Available (was: Open) > Implement Date in ParquetSerde > -- > > Key: HIVE-8119 > URL: https://issues.apache.org/jira/browse/HIVE-8119 > Project: Hive > Issue Type: Sub-task >Reporter: Brock Noland >Assignee: Mohit Sabharwal > Attachments: HIVE-8119.patch > > > Date type in Parquet is discussed here: > http://mail-archives.apache.org/mod_mbox/incubator-parquet-dev/201406.mbox/%3CCAKa9qDkp7xn+H8fNZC7ms3ckd=xr8gdpe7gqgj5o+pybdem...@mail.gmail.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9558) [Parquet] support HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable in vectorized mode
[
https://issues.apache.org/jira/browse/HIVE-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9558:
Attachment: HIVE-9558.1.patch
Thanks for figuring out that! Yes, the failed related case is caused by
{{Execution mode: vectorized}} in test output.
Updated a patch V1 to fix this. Since the Parquet Vectorization is turned off
in HIVE-9235, {{Execution mode: vectorized}} will not be in execution plan. So
the new patch remove it temporarily like HIVE-9235 does. I think we can update
all the output result when turning on Parquet Vectorization.
> [Parquet] support HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable
> in vectorized mode
> ---
>
> Key: HIVE-9558
> URL: https://issues.apache.org/jira/browse/HIVE-9558
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9558.1.patch, HIVE-9558.patch
>
>
> When using Parquet in vectorized mode,
> {{VectorColumnAssignFactory.buildAssigners(..)}} does not handle
> HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable.
> We need fix this and add test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9558) [Parquet] support HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable in vectorized mode
[
https://issues.apache.org/jira/browse/HIVE-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9558:
Status: Patch Available (was: Open)
> [Parquet] support HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable
> in vectorized mode
> ---
>
> Key: HIVE-9558
> URL: https://issues.apache.org/jira/browse/HIVE-9558
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9558.patch
>
>
> When using Parquet in vectorized mode,
> {{VectorColumnAssignFactory.buildAssigners(..)}} does not handle
> HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable.
> We need fix this and add test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9558) [Parquet] support HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable in vectorized mode
[
https://issues.apache.org/jira/browse/HIVE-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9558:
Attachment: HIVE-9558.patch
Upload a patch to add a test case verifying parquet data types in vectorized
mode, and fix failed decimal type.
> [Parquet] support HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable
> in vectorized mode
> ---
>
> Key: HIVE-9558
> URL: https://issues.apache.org/jira/browse/HIVE-9558
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9558.patch
>
>
> When using Parquet in vectorized mode,
> {{VectorColumnAssignFactory.buildAssigners(..)}} does not handle
> HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable.
> We need fix this and add test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9558) [Parquet] support HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable in vectorized mode
Dong Chen created HIVE-9558:
---
Summary: [Parquet] support HiveDecimalWritable, HiveCharWritable,
HiveVarcharWritable in vectorized mode
Key: HIVE-9558
URL: https://issues.apache.org/jira/browse/HIVE-9558
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
When using Parquet in vectorized mode,
{{VectorColumnAssignFactory.buildAssigners(..)}} does not handle
HiveDecimalWritable, HiveCharWritable, HiveVarcharWritable.
We need fix this and add test.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9450) [Parquet] Check all data types work for Parquet in Group By operator
[ https://issues.apache.org/jira/browse/HIVE-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295300#comment-14295300 ] Dong Chen commented on HIVE-9450: - Thank you [~brocknoland], [~Ferd] for your review! > [Parquet] Check all data types work for Parquet in Group By operator > > > Key: HIVE-9450 > URL: https://issues.apache.org/jira/browse/HIVE-9450 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9450.patch, HIVE-9450.patch > > > Check all data types work for Parquet in Group By operator. > 1. Add test cases for data types. > 2. Fix the ClassCastException bug for CHAR&VARCHAR used in group by for > Parquet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9253) MetaStore server should support timeout for long running requests
[
https://issues.apache.org/jira/browse/HIVE-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9253:
Attachment: HIVE-9253.4.patch
Update V4 to address RB comments. Thank you [~leftylev], [~brocknoland] for
your review and feedback.
With regard to client setting the timeout value, I left some reply comments in
RB.
A {{SessionPropertiesListener}} is added for handling client requesting timeout
change. Client could use {{set
metaconf:hive.metastore.server.running.method.timeout 500s}} to change timeout.
If this solution is ok, we may need to document it for user.
> MetaStore server should support timeout for long running requests
> -
>
> Key: HIVE-9253
> URL: https://issues.apache.org/jira/browse/HIVE-9253
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9253.1.patch, HIVE-9253.2.patch, HIVE-9253.2.patch,
> HIVE-9253.3.patch, HIVE-9253.4.patch, HIVE-9253.patch
>
>
> In the description of HIVE-7195, one issue is that MetaStore client timeout
> is quite dumb. The client will timeout and the server has no idea the client
> is gone.
> The server should support timeout when the request from client runs a long
> time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9442) Make sure all data types work for PARQUET
[ https://issues.apache.org/jira/browse/HIVE-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14294628#comment-14294628 ] Dong Chen commented on HIVE-9442: - some sub-tasks are found by testing but not related with data types. > Make sure all data types work for PARQUET > - > > Key: HIVE-9442 > URL: https://issues.apache.org/jira/browse/HIVE-9442 > Project: Hive > Issue Type: Improvement >Reporter: Dong Chen >Assignee: Dong Chen > > In HIVE-9235 (Turn off Parquet Vectorization until all data types work: > DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR), some data types were found not > work for PARQUET. > Work in this Jira will find the root cause, fix it, and add tests for them. > This is an umbrella JIRA. Use sub-tasks for adding tests or fixing bugs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9491) [Parquet] Support Analyze Table with partial scan
Dong Chen created HIVE-9491:
---
Summary: [Parquet] Support Analyze Table with partial scan
Key: HIVE-9491
URL: https://issues.apache.org/jira/browse/HIVE-9491
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
It is nice to have for Parquet to support
{{analyze table table_name compute statistics partialscan;}}, like HIVE-3958
did.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9490) [Parquet] Support Alter Table/Partition Concatenate
[
https://issues.apache.org/jira/browse/HIVE-9490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9490:
Attachment: HIVE-9490.patch-testcase
Add a test for parquet alter table/partition merge. Will update patch with
fixing later.
> [Parquet] Support Alter Table/Partition Concatenate
> ---
>
> Key: HIVE-9490
> URL: https://issues.apache.org/jira/browse/HIVE-9490
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9490.patch-testcase
>
>
> Parquet should support
> {{ALTER TABLE table_name \[PARTITION (partition_key = 'partition_value')\]
> CONCATENATE;}}
> If the table or partition contains many small Parquet files, then the above
> command will merge them into larger files. The merge should happen at row
> group level thereby avoiding the overhead of decompressing and decoding the
> data.
> It is only supported by RCFiles or ORCFiles now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9490) [Parquet] Support Alter Table/Partition Concatenate
Dong Chen created HIVE-9490:
---
Summary: [Parquet] Support Alter Table/Partition Concatenate
Key: HIVE-9490
URL: https://issues.apache.org/jira/browse/HIVE-9490
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Parquet should support
{{ALTER TABLE table_name \[PARTITION (partition_key = 'partition_value')\]
CONCATENATE;}}
If the table or partition contains many small Parquet files, then the above
command will merge them into larger files. The merge should happen at row group
level thereby avoiding the overhead of decompressing and decoding the data.
It is only supported by RCFiles or ORCFiles now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9470) Use a generic writable object to run ColumnaStorageBench write/read tests
[ https://issues.apache.org/jira/browse/HIVE-9470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292872#comment-14292872 ] Dong Chen commented on HIVE-9470: - LGTM. +1 pending test. > Use a generic writable object to run ColumnaStorageBench write/read tests > -- > > Key: HIVE-9470 > URL: https://issues.apache.org/jira/browse/HIVE-9470 > Project: Hive > Issue Type: Improvement >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-9470.1.patch > > > The ColumnarStorageBench benchmark class is using a Parquet writable object > to run all write/read/serialize/deserialize tests. It would be better to use > a more generic writable object (like text writables) to get better benchmark > results between format storages. > Using parquet writables may add advantage when writing parquet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9450) [Parquet] Check all data types work for Parquet in Group By operator
[ https://issues.apache.org/jira/browse/HIVE-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9450: Attachment: HIVE-9450.patch Reattach the patch to see the test result > [Parquet] Check all data types work for Parquet in Group By operator > > > Key: HIVE-9450 > URL: https://issues.apache.org/jira/browse/HIVE-9450 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9450.patch, HIVE-9450.patch > > > Check all data types work for Parquet in Group By operator. > 1. Add test cases for data types. > 2. Fix the ClassCastException bug for CHAR&VARCHAR used in group by for > Parquet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9442) Make sure all data types work for PARQUET
[ https://issues.apache.org/jira/browse/HIVE-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9442: Description: In HIVE-9235 (Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR), some data types were found not work for PARQUET. Work in this Jira will find the root cause, fix it, and add tests for them. This is an umbrella JIRA. Use sub-tasks for adding tests or fixing bugs. was: In HIVE-9235 (Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR), some data types were found not work for PARQUET. Work in this Jira will find the root cause, fix it, and add tests for them. > Make sure all data types work for PARQUET > - > > Key: HIVE-9442 > URL: https://issues.apache.org/jira/browse/HIVE-9442 > Project: Hive > Issue Type: Improvement >Reporter: Dong Chen >Assignee: Dong Chen > > In HIVE-9235 (Turn off Parquet Vectorization until all data types work: > DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR), some data types were found not > work for PARQUET. > Work in this Jira will find the root cause, fix it, and add tests for them. > This is an umbrella JIRA. Use sub-tasks for adding tests or fixing bugs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-9442) Make sure all data types work for PARQUET
[ https://issues.apache.org/jira/browse/HIVE-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-9442 started by Dong Chen. --- > Make sure all data types work for PARQUET > - > > Key: HIVE-9442 > URL: https://issues.apache.org/jira/browse/HIVE-9442 > Project: Hive > Issue Type: Improvement >Reporter: Dong Chen >Assignee: Dong Chen > > In HIVE-9235 (Turn off Parquet Vectorization until all data types work: > DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR), some data types were found not > work for PARQUET. > Work in this Jira will find the root cause, fix it, and add tests for them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9450) [Parquet] Check all data types work for Parquet in Group By operator
[ https://issues.apache.org/jira/browse/HIVE-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9450: Status: Patch Available (was: Open) > [Parquet] Check all data types work for Parquet in Group By operator > > > Key: HIVE-9450 > URL: https://issues.apache.org/jira/browse/HIVE-9450 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9450.patch > > > Check all data types work for Parquet in Group By operator. > 1. Add test cases for data types. > 2. Fix the ClassCastException bug for CHAR&VARCHAR used in group by for > Parquet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9450) [Parquet] Check all data types work for Parquet in Group By operator
[
https://issues.apache.org/jira/browse/HIVE-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9450:
Attachment: HIVE-9450.patch
Upload a patch to add tests for data types used in group by operator, and fix
CHAR/VARCHAR bug.
Root cause is in {{WritableHiveCharObjectInspector.copyObject()}}. As described
in HIVE-8205, the object inspector should handle the conversion of Text
writable. We add this special handling here. Same for
{{WritableHiveVarcharObjectInspector.copyObject()}}.
> [Parquet] Check all data types work for Parquet in Group By operator
>
>
> Key: HIVE-9450
> URL: https://issues.apache.org/jira/browse/HIVE-9450
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9450.patch
>
>
> Check all data types work for Parquet in Group By operator.
> 1. Add test cases for data types.
> 2. Fix the ClassCastException bug for CHAR&VARCHAR used in group by for
> Parquet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9450) [Parquet] Check all data types work for Parquet in Group By operator
Dong Chen created HIVE-9450: --- Summary: [Parquet] Check all data types work for Parquet in Group By operator Key: HIVE-9450 URL: https://issues.apache.org/jira/browse/HIVE-9450 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Check all data types work for Parquet in Group By operator. 1. Add test cases for data types. 2. Fix the ClassCastException bug for CHAR&VARCHAR used in group by for Parquet. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9442) Make sure all data types work for PARQUET
[ https://issues.apache.org/jira/browse/HIVE-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289424#comment-14289424 ] Dong Chen commented on HIVE-9442: - Work procedure is: 1. run existed file format(ORC, etc) test cases for Parquet non-vectorized. Fix bugs and add tests if any. 2. run existed vectorized(ORC) test cases for Parquet. Fix bugs and add tests if any. (The on-going Parquet Vectorization work may introduce change when it's done. But I think we can still do this check now. They are relatively independent and we could see and make change at that time.) 3. make the coverage of vectorized test cases for Parquet about data types compelte. The goal is to make all data types (except DATE, not implemented yet) work for non-vectorized & vectorized Parquet. > Make sure all data types work for PARQUET > - > > Key: HIVE-9442 > URL: https://issues.apache.org/jira/browse/HIVE-9442 > Project: Hive > Issue Type: Improvement >Reporter: Dong Chen >Assignee: Dong Chen > > In HIVE-9235 (Turn off Parquet Vectorization until all data types work: > DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR), some data types were found not > work for PARQUET. > Work in this Jira will find the root cause, fix it, and add tests for them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9442) Make sure all data types work for PARQUET
[ https://issues.apache.org/jira/browse/HIVE-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9442: Issue Type: Improvement (was: Sub-task) Parent: (was: HIVE-8120) > Make sure all data types work for PARQUET > - > > Key: HIVE-9442 > URL: https://issues.apache.org/jira/browse/HIVE-9442 > Project: Hive > Issue Type: Improvement >Reporter: Dong Chen >Assignee: Dong Chen > > In HIVE-9235 (Turn off Parquet Vectorization until all data types work: > DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR), some data types were found not > work for PARQUET. > Work in this Jira will find the root cause, fix it, and add tests for them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9434) Shim the method Path.getPathWithoutSchemeAndAuthority
[
https://issues.apache.org/jira/browse/HIVE-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14288918#comment-14288918
]
Dong Chen commented on HIVE-9434:
-
Failed tests seem not related.
> Shim the method Path.getPathWithoutSchemeAndAuthority
> -
>
> Key: HIVE-9434
> URL: https://issues.apache.org/jira/browse/HIVE-9434
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 0.15.0
>Reporter: Brock Noland
>Assignee: Dong Chen
> Fix For: 0.15.0
>
> Attachments: HIVE-9434.patch
>
>
> Since Hadoop 1 does not have the method
> {{Path.getPathWithoutSchemeAndAuthority}} we need to shim it out.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9434) Shim the method Path.getPathWithoutSchemeAndAuthority
[
https://issues.apache.org/jira/browse/HIVE-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9434:
Attachment: HIVE-9434.patch
Upload a patch to shim the method {{Path.getPathWithoutSchemeAndAuthority}}
> Shim the method Path.getPathWithoutSchemeAndAuthority
> -
>
> Key: HIVE-9434
> URL: https://issues.apache.org/jira/browse/HIVE-9434
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 0.15.0
>Reporter: Brock Noland
>Assignee: Dong Chen
> Fix For: 0.15.0
>
> Attachments: HIVE-9434.patch
>
>
> Since Hadoop 1 does not have the method
> {{Path.getPathWithoutSchemeAndAuthority}} we need to shim it out.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9434) Shim the method Path.getPathWithoutSchemeAndAuthority
[
https://issues.apache.org/jira/browse/HIVE-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9434:
Status: Patch Available (was: Open)
> Shim the method Path.getPathWithoutSchemeAndAuthority
> -
>
> Key: HIVE-9434
> URL: https://issues.apache.org/jira/browse/HIVE-9434
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 0.15.0
>Reporter: Brock Noland
>Assignee: Dong Chen
> Fix For: 0.15.0
>
>
> Since Hadoop 1 does not have the method
> {{Path.getPathWithoutSchemeAndAuthority}} we need to shim it out.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9253) MetaStore server should support timeout for long running requests
[ https://issues.apache.org/jira/browse/HIVE-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14287083#comment-14287083 ] Dong Chen commented on HIVE-9253: - Thanks for your review! [~leftylev]. Adding the word "long" is a good idea. This patch V3 was trying to fix the failed test cases. I will rename the parameter in V4. > MetaStore server should support timeout for long running requests > - > > Key: HIVE-9253 > URL: https://issues.apache.org/jira/browse/HIVE-9253 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9253.1.patch, HIVE-9253.2.patch, HIVE-9253.2.patch, > HIVE-9253.3.patch, HIVE-9253.patch > > > In the description of HIVE-7195, one issue is that MetaStore client timeout > is quite dumb. The client will timeout and the server has no idea the client > is gone. > The server should support timeout when the request from client runs a long > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9253) MetaStore server should support timeout for long running requests
[ https://issues.apache.org/jira/browse/HIVE-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9253: Attachment: HIVE-9253.3.patch Update v3 > MetaStore server should support timeout for long running requests > - > > Key: HIVE-9253 > URL: https://issues.apache.org/jira/browse/HIVE-9253 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9253.1.patch, HIVE-9253.2.patch, HIVE-9253.2.patch, > HIVE-9253.3.patch, HIVE-9253.patch > > > In the description of HIVE-7195, one issue is that MetaStore client timeout > is quite dumb. The client will timeout and the server has no idea the client > is gone. > The server should support timeout when the request from client runs a long > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9442) Make sure all data types work for PARQUET
Dong Chen created HIVE-9442: --- Summary: Make sure all data types work for PARQUET Key: HIVE-9442 URL: https://issues.apache.org/jira/browse/HIVE-9442 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen In HIVE-9235 (Turn off Parquet Vectorization until all data types work: DECIMAL, DATE, TIMESTAMP, CHAR, and VARCHAR), some data types were found not work for PARQUET. Work in this Jira will find the root cause, fix it, and add tests for them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-9434) Shim the method Path.getPathWithoutSchemeAndAuthority
[
https://issues.apache.org/jira/browse/HIVE-9434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen reassigned HIVE-9434:
---
Assignee: Dong Chen
> Shim the method Path.getPathWithoutSchemeAndAuthority
> -
>
> Key: HIVE-9434
> URL: https://issues.apache.org/jira/browse/HIVE-9434
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 0.15.0
>Reporter: Brock Noland
>Assignee: Dong Chen
> Fix For: 0.15.0
>
>
> Since Hadoop 1 does not have the method
> {{Path.getPathWithoutSchemeAndAuthority}} we need to shim it out.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9253) MetaStore server should support timeout for long running requests
[
https://issues.apache.org/jira/browse/HIVE-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9253:
Attachment: HIVE-9253.2.patch
Thanks very much for your help! [~brocknoland]
I updated the patch V2 with all changes for review. The 3 steps are all done.
1. Implement a threadlocal {{RuntimeTimeout}} class
2. add {{RuntimeTimeout.checkTimeout()}} in HMS code. Added some check in
{{MetaStoreDirectSql}}, which can serve as an example. Since the places needing
check (most are probable in {{ObjectStore}} or {{MetaStoreDirectSql}}) might be
a lot, how about we find them and add checking in a follow-on Jira?
3. Client could reset the timeout value. I found reuse the setMetaConf() api
and add a listener might be ok, instead of creating a new request. This reuse
could avoid adding new interface in HS2 for user to change timeout value.
> MetaStore server should support timeout for long running requests
> -
>
> Key: HIVE-9253
> URL: https://issues.apache.org/jira/browse/HIVE-9253
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9253.1.patch, HIVE-9253.2.patch, HIVE-9253.patch
>
>
> In the description of HIVE-7195, one issue is that MetaStore client timeout
> is quite dumb. The client will timeout and the server has no idea the client
> is gone.
> The server should support timeout when the request from client runs a long
> time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9264) Merge encryption branch to trunk
[ https://issues.apache.org/jira/browse/HIVE-9264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14282223#comment-14282223 ] Dong Chen commented on HIVE-9264: - +1 Recent change also looks good!! Thanks for the patch and merge! [~brocknoland] > Merge encryption branch to trunk > > > Key: HIVE-9264 > URL: https://issues.apache.org/jira/browse/HIVE-9264 > Project: Hive > Issue Type: Sub-task >Affects Versions: 0.15.0 >Reporter: Brock Noland >Assignee: Brock Noland > Attachments: HIVE-9264.1.patch, HIVE-9264.2.patch, HIVE-9264.2.patch, > HIVE-9264.2.patch, HIVE-9264.3.patch, HIVE-9264.3.patch, HIVE-9264.3.patch > > > The team working on the encryption branch would like to merge their work to > trunk. This jira will track that effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9356) Fail to handle the case that a qfile contains a semicolon in the annotation
[ https://issues.apache.org/jira/browse/HIVE-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9356: Attachment: HIVE-9356.2-encryption.patch Right! I missed the HiveCommand only for testing. Thank your [~spena] Update patch V2 > Fail to handle the case that a qfile contains a semicolon in the annotation > --- > > Key: HIVE-9356 > URL: https://issues.apache.org/jira/browse/HIVE-9356 > Project: Hive > Issue Type: Sub-task >Affects Versions: encryption-branch >Reporter: Ferdinand Xu >Assignee: Dong Chen > Fix For: encryption-branch > > Attachments: HIVE-9356-encryption.patch, > HIVE-9356.1-encryption.patch, HIVE-9356.2-encryption.patch, HIVE-9356.patch > > > Currently, we split the qfile by the semicolon. It should be able to handle > the comment statement in the qfile with a semicolon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9356) Fail to handle the case that a qfile contains a semicolon in the annotation
[
https://issues.apache.org/jira/browse/HIVE-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9356:
Attachment: HIVE-9356.1-encryption.patch
Thanks for your review! Sergio, Brock.
I found we have to handle below 2 difference cases of .q in current command
parser implementation, so updated patch V1.
{quote}
--comment;
sql statement
{quote}
and
{quote}
--comment;
HiveCommand
{quote}
> Fail to handle the case that a qfile contains a semicolon in the annotation
> ---
>
> Key: HIVE-9356
> URL: https://issues.apache.org/jira/browse/HIVE-9356
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: encryption-branch
>Reporter: Ferdinand Xu
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9356-encryption.patch,
> HIVE-9356.1-encryption.patch, HIVE-9356.patch
>
>
> Currently, we split the qfile by the semicolon. It should be able to handle
> the comment statement in the qfile with a semicolon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9356) Fail to handle the case that a qfile contains a semicolon
[ https://issues.apache.org/jira/browse/HIVE-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9356: Attachment: HIVE-9356-encryption.patch Rename patch > Fail to handle the case that a qfile contains a semicolon > - > > Key: HIVE-9356 > URL: https://issues.apache.org/jira/browse/HIVE-9356 > Project: Hive > Issue Type: Sub-task >Affects Versions: encryption-branch >Reporter: Ferdinand Xu >Assignee: Dong Chen > Fix For: encryption-branch > > Attachments: HIVE-9356-encryption.patch, HIVE-9356.patch > > > Currently, we split the qfile by the semicolon. It should be able to handle > the comment statement in the qfile with a semicolon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9356) Fail to handle the case that a qfile contains a semicolon
[ https://issues.apache.org/jira/browse/HIVE-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9356: Attachment: HIVE-9356.patch > Fail to handle the case that a qfile contains a semicolon > - > > Key: HIVE-9356 > URL: https://issues.apache.org/jira/browse/HIVE-9356 > Project: Hive > Issue Type: Sub-task >Affects Versions: encryption-branch >Reporter: Ferdinand Xu >Assignee: Dong Chen > Fix For: encryption-branch > > Attachments: HIVE-9356.patch > > > Currently, we split the qfile by the semicolon. It should be able to handle > the comment statement in the qfile with a semicolon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9356) Fail to handle the case that a qfile contains a semicolon
[
https://issues.apache.org/jira/browse/HIVE-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14274647#comment-14274647
]
Dong Chen commented on HIVE-9356:
-
A small fix for the failed case {{encryption_load_data_to_encrypted_tables.q}}
in
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-ENCRYPTION-Build/2/#showFailuresLink
Make the command parser in {{QTestUtil.executeClientInternal()}} could handle
comments (start with "--")
> Fail to handle the case that a qfile contains a semicolon
> -
>
> Key: HIVE-9356
> URL: https://issues.apache.org/jira/browse/HIVE-9356
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: encryption-branch
>Reporter: Ferdinand Xu
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9356.patch
>
>
> Currently, we split the qfile by the semicolon. It should be able to handle
> the comment statement in the qfile with a semicolon.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9356) Fail to handle the case that a qfile contains a semicolon
[ https://issues.apache.org/jira/browse/HIVE-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9356: Status: Patch Available (was: Open) > Fail to handle the case that a qfile contains a semicolon > - > > Key: HIVE-9356 > URL: https://issues.apache.org/jira/browse/HIVE-9356 > Project: Hive > Issue Type: Sub-task >Affects Versions: encryption-branch >Reporter: Ferdinand Xu >Assignee: Dong Chen > Fix For: encryption-branch > > > Currently, we split the qfile by the semicolon. It should be able to handle > the comment statement in the qfile with a semicolon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9356) Fail to handle the case that a qfile contains a semicolon
[ https://issues.apache.org/jira/browse/HIVE-9356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9356: Description: Currently, we split the qfile by the semicolon. It should be able to handle the comment statement in the qfile with a semicolon. (was: Currently, we split the qfile by the semicolon. It should be avoid to end up a statement in the qfile with a semicolon.) > Fail to handle the case that a qfile contains a semicolon > - > > Key: HIVE-9356 > URL: https://issues.apache.org/jira/browse/HIVE-9356 > Project: Hive > Issue Type: Sub-task >Affects Versions: encryption-branch >Reporter: Ferdinand Xu >Assignee: Dong Chen > Fix For: encryption-branch > > > Currently, we split the qfile by the semicolon. It should be able to handle > the comment statement in the qfile with a semicolon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9253) MetaStore server should support timeout for long running requests
[
https://issues.apache.org/jira/browse/HIVE-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273737#comment-14273737
]
Dong Chen commented on HIVE-9253:
-
When unit testing the changes in {{RetryingHMSHandler}}, I found it is hard to
simulate the method timeout case. I could manually debug and pause the method
so that timeout, and the {{checkTimeout()}} in usage 2 work fine.
However, I did not find a way to write a stable case for it...
Not sure using a specific data set for the case is ok, since the speed may vary
in build machines. And the cases like {{TestHiveMetaStore}} focusing on
client/server layer seems cannot help.
Any ideas are highly appreciated! :)
> MetaStore server should support timeout for long running requests
> -
>
> Key: HIVE-9253
> URL: https://issues.apache.org/jira/browse/HIVE-9253
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9253.1.patch, HIVE-9253.patch
>
>
> In the description of HIVE-7195, one issue is that MetaStore client timeout
> is quite dumb. The client will timeout and the server has no idea the client
> is gone.
> The server should support timeout when the request from client runs a long
> time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9253) MetaStore server should support timeout for long running requests
[ https://issues.apache.org/jira/browse/HIVE-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9253: Attachment: HIVE-9253.1.patch > MetaStore server should support timeout for long running requests > - > > Key: HIVE-9253 > URL: https://issues.apache.org/jira/browse/HIVE-9253 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9253.1.patch, HIVE-9253.patch > > > In the description of HIVE-7195, one issue is that MetaStore client timeout > is quite dumb. The client will timeout and the server has no idea the client > is gone. > The server should support timeout when the request from client runs a long > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9253) MetaStore server should support timeout for long running requests
[
https://issues.apache.org/jira/browse/HIVE-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14273722#comment-14273722
]
Dong Chen commented on HIVE-9253:
-
Thank you [~brocknoland] for your great idea!
I updated a patch V1 implementing the step 1 in below description for early
feedback. Could you take a look at it? Thanks.
RB: https://reviews.apache.org/r/29807/
*Step 1 (in patch)*: Implement a threadlocal {{RuntimeTimeout}} class.
It will register when a client connection thread start, and monitor every
method call of {{HiveMetaStore.HMSHandler}}. Its timeout value will be init by
conf and reset by client if wanted.
In the patch, 2 usage of starting the timer is shown. Do you think usage 2 is
OK?
* usage 1 start the timer in one place and is generic. But it seems threadlocal
works unhappily with reflection. When invoking
{{RuntimeTimeout.checkTimeout()}} in a method like {{get_table}}, the
{{RuntimeTimeout.startTime}} is not set as expected in
{{RuntimeTimeout.startCounting}} and still -1. (still trying to root cause it..)
* usage 2 needs to start timer in every method we want to track. Its sunny side
is, {{startCounting()}} and {{checkTimeout()}} works fine in this approach by
verifying manually.
*Step 2*: Add {{RuntimeTimeout.checkTimeout()}} in proper place.
I think the places could be 2 type:
* after a probable long time database query
* inner loop
And the involved classes / methods might be:
* In methods of {{ObjectStore}}, such as getPartitionsInternal
* In methods of {{MetaStoreDirectSql}}, such as getPartitionsFromPartitionIds
*Step 3*: Clients could set the timeout value as they want
* create a new request {{SetSessionPropertiesRequest}} is a good idea. Server
will handle this msg and invoke {{RuntimeTimeout.resetTimeout()}}.
* Maybe session-level.
* Since this is relatively independent, I put it in the last step.
> MetaStore server should support timeout for long running requests
> -
>
> Key: HIVE-9253
> URL: https://issues.apache.org/jira/browse/HIVE-9253
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9253.1.patch, HIVE-9253.patch
>
>
> In the description of HIVE-7195, one issue is that MetaStore client timeout
> is quite dumb. The client will timeout and the server has no idea the client
> is gone.
> The server should support timeout when the request from client runs a long
> time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270486#comment-14270486
]
Dong Chen commented on HIVE-9307:
-
Not necessary. Other .q files result keep same.
The trick is that:
In {{QTestUtil.planMask}}, one is {{LOCATION}}. This cause the table location
in create query in PREHOOK / POSTHOOK masked.
In the 2 .q file of this patch, it is {{location}} so that it is not masked, as
expected.
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9307.1.patch, HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9307:
Fix Version/s: encryption-branch
Status: Patch Available (was: Open)
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9307.1.patch, HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9307:
Attachment: HIVE-9307.1.patch
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9307.1.patch, HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use metastore warehouse dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270398#comment-14270398
]
Dong Chen commented on HIVE-9307:
-
Update patch V1.
The variable is used in 3 statement: {{CREATE table sql}}, {{dfs command}}, and
{{CRYPTO command}}. The first 2 already can handle this variable. For CRYPTO
command, since we only use it for test, I added the handling in
{{QTestUtil.executeTestCommand()}} for simplicity.
The output of 2 cases changed by 2 lines, which are the table location in
create query in PREHOOK / POSTHOOK is masked. After double check, I think it is
OK and I misunderstood before.
> Use metastore warehouse dir variable from conf instead of hard coded dir in
> encryption test
> ---
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9253) MetaStore server should support timeout for long running requests
[ https://issues.apache.org/jira/browse/HIVE-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9253: Summary: MetaStore server should support timeout for long running requests (was: MetaStore server should detect client timeout) > MetaStore server should support timeout for long running requests > - > > Key: HIVE-9253 > URL: https://issues.apache.org/jira/browse/HIVE-9253 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9253.patch > > > In the description of HIVE-7195, one issue is that MetaStore client timeout > is quite dumb. The client will timeout and the server has no idea the client > is gone. > The server should support timeout when the request from client runs a long > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9253) MetaStore server should detect client timeout
[ https://issues.apache.org/jira/browse/HIVE-9253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9253: Description: In the description of HIVE-7195, one issue is that MetaStore client timeout is quite dumb. The client will timeout and the server has no idea the client is gone. The server should support timeout when the request from client runs a long time. was: In the description of HIVE-7195, one issue is that MetaStore client timeout is quite dumb. The client will timeout and the server has no idea the client is gone. The server should detect client timeout. > MetaStore server should detect client timeout > - > > Key: HIVE-9253 > URL: https://issues.apache.org/jira/browse/HIVE-9253 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9253.patch > > > In the description of HIVE-7195, one issue is that MetaStore client timeout > is quite dumb. The client will timeout and the server has no idea the client > is gone. > The server should support timeout when the request from client runs a long > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9307) Use MetaStore dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9307:
Attachment: HIVE-9307.patch
> Use MetaStore dir variable from conf instead of hard coded dir in encryption
> test
> -
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9307.patch
>
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9307) Use MetaStore dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9307:
Description:
NO PRECOMMIT TESTS
Use the following variable to get the metastore directory
$\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
was:
Use the following variable to get the metastore directory
$\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
> Use MetaStore dir variable from conf instead of hard coded dir in encryption
> test
> -
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> NO PRECOMMIT TESTS
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9307) Use MetaStore dir variable from conf instead of hard coded dir in encryption test
[
https://issues.apache.org/jira/browse/HIVE-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14269310#comment-14269310
]
Dong Chen commented on HIVE-9307:
-
Patch attached for review. Still not good enough.
When using $\{hiveconf:hive.metastore.warehouse.dir\} instead of
/build/ql/test/data/warehouse, there is a slight difference. The variable value
is {{*hdfs://localhost:37553*/build/ql/test/data/warehouse}}. A schema and
network address is added.
The test still works, but diff failed. The result log changes a little by mask
pattern. But I think the masked line is useful and should not be masked. Still
thinking a better solution to fix this.
In order not to impact our branch test today, let's not merge it for now.
> Use MetaStore dir variable from conf instead of hard coded dir in encryption
> test
> -
>
> Key: HIVE-9307
> URL: https://issues.apache.org/jira/browse/HIVE-9307
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> Use the following variable to get the metastore directory
> $\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9297) HS2 should be able to write query logs to HDFS or at least copy them there
[
https://issues.apache.org/jira/browse/HIVE-9297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14269295#comment-14269295
]
Dong Chen commented on HIVE-9297:
-
A good idea! It would be great I could help on this. But does not have enough
cycle recently.
Mark it. When I get some time later and if this issue is still open, I will
take it.
BTW. As a first thought, I think the archive code might start at
{{o.a.h.h.ql.session.OperationLog.LogFile.remove()}} before deleting log and
add some operation context info for archiving.
> HS2 should be able to write query logs to HDFS or at least copy them there
> --
>
> Key: HIVE-9297
> URL: https://issues.apache.org/jira/browse/HIVE-9297
> Project: Hive
> Issue Type: Task
>Reporter: Brock Noland
>
> In HIVE-4629 [~dongc] did some fantastic work to support log retrieval. I
> think a good next step is to archive these logs in HDFS. This can be done via
> by writing them directly to HDFS or copying them to HDFS when the query
> completes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268836#comment-14268836
]
Dong Chen commented on HIVE-9283:
-
Good idea!. Thanks for your review, [~spena], [~brocknoland]. I filed HIVE-9307
to track this.
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9283.patch
>
>
> NO PRECOMMIT TESTS
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key. (Put in HIVE-9286)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9307) Use MetaStore dir variable from conf instead of hard coded dir in encryption test
Dong Chen created HIVE-9307:
---
Summary: Use MetaStore dir variable from conf instead of hard
coded dir in encryption test
Key: HIVE-9307
URL: https://issues.apache.org/jira/browse/HIVE-9307
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Use the following variable to get the metastore directory
$\{hiveconf:hive.metastore.warehouse.dir\} in test cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9286) Add clean up code in some encryption related test cases
[ https://issues.apache.org/jira/browse/HIVE-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267516#comment-14267516 ] Dong Chen commented on HIVE-9286: - Since HIVE-9261 has already been created for updating test result change (.out file), this patch only updated .q files. > Add clean up code in some encryption related test cases > --- > > Key: HIVE-9286 > URL: https://issues.apache.org/jira/browse/HIVE-9286 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Fix For: encryption-branch > > Attachments: HIVE-9286.patch > > > NO PRECOMMIT TESTS > Add cleanup at the end of .q files: drop table and delete key. > Otherwise, some cases will fail caused by cannot create existed key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9261) Update the output files for the encryption qtests since the output format changed
[ https://issues.apache.org/jira/browse/HIVE-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9261: Attachment: HIVE-9261.patch This patch updated the output result of encryption test cases. It covers HIVE-9254 (output format change), HIVE-9283 (table location change for not being masked), and HIVE-9286 (change of adding cleanup). > Update the output files for the encryption qtests since the output format > changed > - > > Key: HIVE-9261 > URL: https://issues.apache.org/jira/browse/HIVE-9261 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Dong Chen > Fix For: encryption-branch > > Attachments: HIVE-9261.patch > > > NO PRECOMMIT TESTS > Should update the output files due to the output format changes in HIVE-9254 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9261) Update the output files for the encryption qtests since the output format changed
[ https://issues.apache.org/jira/browse/HIVE-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9261: Fix Version/s: encryption-branch Status: Patch Available (was: Open) > Update the output files for the encryption qtests since the output format > changed > - > > Key: HIVE-9261 > URL: https://issues.apache.org/jira/browse/HIVE-9261 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Dong Chen > Fix For: encryption-branch > > > NO PRECOMMIT TESTS > Should update the output files due to the output format changes in HIVE-9254 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9261) Update the output files for the encryption qtests since the output format changed
[ https://issues.apache.org/jira/browse/HIVE-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9261: Description: NO PRECOMMIT TESTS Should update the output files due to the output format changes in HIVE-9254 was:Should update the output files due to the output format changes in HIVE-9254 > Update the output files for the encryption qtests since the output format > changed > - > > Key: HIVE-9261 > URL: https://issues.apache.org/jira/browse/HIVE-9261 > Project: Hive > Issue Type: Sub-task >Reporter: Ferdinand Xu >Assignee: Dong Chen > > NO PRECOMMIT TESTS > Should update the output files due to the output format changes in HIVE-9254 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9286) Add clean up code in some encryption related test cases
[ https://issues.apache.org/jira/browse/HIVE-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9286: Attachment: HIVE-9286.patch Patch uploaded. Drop table and delete key at the end in the .q files. > Add clean up code in some encryption related test cases > --- > > Key: HIVE-9286 > URL: https://issues.apache.org/jira/browse/HIVE-9286 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Fix For: encryption-branch > > Attachments: HIVE-9286.patch > > > NO PRECOMMIT TESTS > Add cleanup at the end of .q files: drop table and delete key. > Otherwise, some cases will fail caused by cannot create existed key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9286) Add clean up code in some encryption related test cases
[ https://issues.apache.org/jira/browse/HIVE-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9286: Fix Version/s: encryption-branch Status: Patch Available (was: Open) > Add clean up code in some encryption related test cases > --- > > Key: HIVE-9286 > URL: https://issues.apache.org/jira/browse/HIVE-9286 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Fix For: encryption-branch > > > NO PRECOMMIT TESTS > Add cleanup at the end of .q files: drop table and delete key. > Otherwise, some cases will fail caused by cannot create existed key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9286) Add clean up code in some encryption related test cases
[ https://issues.apache.org/jira/browse/HIVE-9286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9286: Description: NO PRECOMMIT TESTS Add cleanup at the end of .q files: drop table and delete key. Otherwise, some cases will fail caused by cannot create existed key. was: Add cleanup at the end of .q files. drop table and delete key. Otherwise, some cases will fail caused by cannot create existed key. > Add clean up code in some encryption related test cases > --- > > Key: HIVE-9286 > URL: https://issues.apache.org/jira/browse/HIVE-9286 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Fix For: encryption-branch > > > NO PRECOMMIT TESTS > Add cleanup at the end of .q files: drop table and delete key. > Otherwise, some cases will fail caused by cannot create existed key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267500#comment-14267500
]
Dong Chen commented on HIVE-9283:
-
Since HIVE-9261 has already been created for updating test result change (.out
file), this patch only updated .q files. This also make review a little easier
without lots of generated .out file.
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9283.patch
>
>
> NO PRECOMMIT TESTS
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key. (Put in HIVE-9286)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9283:
Fix Version/s: encryption-branch
Status: Patch Available (was: Open)
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Fix For: encryption-branch
>
> Attachments: HIVE-9283.patch
>
>
> NO PRECOMMIT TESTS
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key. (Put in HIVE-9286)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9283:
Description:
NO PRECOMMIT TESTS
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key. (Put in HIVE-9286)
was:
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key. (Put in HIVE-9286)
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9283.patch
>
>
> NO PRECOMMIT TESTS
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key. (Put in HIVE-9286)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9283:
Attachment: HIVE-9283.patch
This patch update the table location in the .q files
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
> Attachments: HIVE-9283.patch
>
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key. (Put in HIVE-9286)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9287) Only 3 encryption test cases was run. The test configuration is not correct
[ https://issues.apache.org/jira/browse/HIVE-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9287: Attachment: HIVE-9287.patch A small change attached. The comma is missed, so that some cases is not parsed and added expected. > Only 3 encryption test cases was run. The test configuration is not correct > --- > > Key: HIVE-9287 > URL: https://issues.apache.org/jira/browse/HIVE-9287 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9287.patch > > > NO PRECOMMIT TESTS > Only 3 encryption test cases was run. The test configuration is not correct. > We should add all test case file in testconfiguration.properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9287) Only 3 encryption test cases was run. The test configuration is not correct
[ https://issues.apache.org/jira/browse/HIVE-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9287: Description: NO PRECOMMIT TESTS Only 3 encryption test cases was run. The test configuration is not correct. We should add all test case file in testconfiguration.properties was: Only 3 encryption test cases was run. The test configuration is not correct. We should add all test case file in testconfiguration.properties > Only 3 encryption test cases was run. The test configuration is not correct > --- > > Key: HIVE-9287 > URL: https://issues.apache.org/jira/browse/HIVE-9287 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > > NO PRECOMMIT TESTS > Only 3 encryption test cases was run. The test configuration is not correct. > We should add all test case file in testconfiguration.properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9287) Only 3 encryption test cases was run. The test configuration is not correct
[ https://issues.apache.org/jira/browse/HIVE-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9287: Status: Patch Available (was: Open) > Only 3 encryption test cases was run. The test configuration is not correct > --- > > Key: HIVE-9287 > URL: https://issues.apache.org/jira/browse/HIVE-9287 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > > Only 3 encryption test cases was run. The test configuration is not correct. > We should add all test case file in testconfiguration.properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9287) Only 3 encryption test cases was run. The test configuration is not correct
Dong Chen created HIVE-9287: --- Summary: Only 3 encryption test cases was run. The test configuration is not correct Key: HIVE-9287 URL: https://issues.apache.org/jira/browse/HIVE-9287 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Only 3 encryption test cases was run. The test configuration is not correct. We should add all test case file in testconfiguration.properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9283:
Description:
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key. (Put in HIVE-9286)
was:
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key.
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key. (Put in HIVE-9286)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267361#comment-14267361
]
Dong Chen commented on HIVE-9283:
-
Sorry for typo... It is HIVE-9286
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267360#comment-14267360
]
Dong Chen commented on HIVE-9283:
-
Sorry for typo... It is HIVE-9286
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14267357#comment-14267357
]
Dong Chen commented on HIVE-9283:
-
Good suggestion! This can make the piece of code focused, and easy to revert or
review. Thank [~Ferd]
I put the 2nd point in HIVE-9268.
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9286) Add clean up code in some encryption related test cases
Dong Chen created HIVE-9286: --- Summary: Add clean up code in some encryption related test cases Key: HIVE-9286 URL: https://issues.apache.org/jira/browse/HIVE-9286 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Add cleanup at the end of .q files. drop table and delete key. Otherwise, some cases will fail caused by cannot create existed key. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9283) Improve encryption related test cases
[
https://issues.apache.org/jira/browse/HIVE-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9283:
Description:
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key.
was:
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key.
> Improve encryption related test cases
> -
>
> Key: HIVE-9283
> URL: https://issues.apache.org/jira/browse/HIVE-9283
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>
> I found some test cases .q file could be improved by:
> 1. change the table location from {{/user/hive/warehouse...}} to
> {{/build/ql/test/data/warehouse/...}}.
> The reason is that the default warehouse dir defined in QTestUtil is the
> latter one, and the partial mask is based on it. I think it is better to make
> test cases consistent with code. Also the .hive_staging location we want in
> .out will be shown then.
> 2. add cleanup at the end.
> drop table and delete key. Otherwise, some cases will fail caused by cannot
> create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9283) Improve encryption related test cases
Dong Chen created HIVE-9283:
---
Summary: Improve encryption related test cases
Key: HIVE-9283
URL: https://issues.apache.org/jira/browse/HIVE-9283
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
I found some test cases .q file could be improved by:
1. change the table location from {{/user/hive/warehouse...}} to
{{/build/ql/test/data/warehouse/...}}.
The reason is that the default warehouse dir defined in QTestUtil is the latter
one, and the partial mask is based on it. I think it is better to make test
cases consistent with code. Also the .hive_staging location we want in .out
will be shown then.
2. add cleanup at the end.
drop table and delete key. Otherwise, some cases will fail caused by cannot
create existed key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9280) The Arguments of CRYPTO command is not parsed correctly in QTestUtil.executeTestCommand()
[ https://issues.apache.org/jira/browse/HIVE-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9280: Attachment: HIVE-9280.patch A small fix. The command might start with \n, so substring it without trim first might index incorrectly. > The Arguments of CRYPTO command is not parsed correctly in > QTestUtil.executeTestCommand() > - > > Key: HIVE-9280 > URL: https://issues.apache.org/jira/browse/HIVE-9280 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > Attachments: HIVE-9280.patch > > > NO PRECOMMIT TESTS > The Arguments of CRYPTO command is not parsed correctly in > QTestUtil.executeTestCommand(). It is found after fix HIVE-9279. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9280) The Arguments of CRYPTO command is not parsed correctly in QTestUtil.executeTestCommand()
[ https://issues.apache.org/jira/browse/HIVE-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Chen updated HIVE-9280: Status: Patch Available (was: Open) > The Arguments of CRYPTO command is not parsed correctly in > QTestUtil.executeTestCommand() > - > > Key: HIVE-9280 > URL: https://issues.apache.org/jira/browse/HIVE-9280 > Project: Hive > Issue Type: Sub-task >Reporter: Dong Chen >Assignee: Dong Chen > > NO PRECOMMIT TESTS > The Arguments of CRYPTO command is not parsed correctly in > QTestUtil.executeTestCommand(). It is found after fix HIVE-9279. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9280) The Arguments of CRYPTO command is not parsed correctly in QTestUtil.executeTestCommand()
Dong Chen created HIVE-9280: --- Summary: The Arguments of CRYPTO command is not parsed correctly in QTestUtil.executeTestCommand() Key: HIVE-9280 URL: https://issues.apache.org/jira/browse/HIVE-9280 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen NO PRECOMMIT TESTS The Arguments of CRYPTO command is not parsed correctly in QTestUtil.executeTestCommand(). It is found after fix HIVE-9279. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9279) TestEncryptedHDFSCliDriver get exception "Could not execute test command" for encryption test cases
[
https://issues.apache.org/jira/browse/HIVE-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9279:
Attachment: HIVE-9279.patch
Uploaded patch. Should trim the command string to get rid of '\n'.
> TestEncryptedHDFSCliDriver get exception "Could not execute test command" for
> encryption test cases
> ---
>
> Key: HIVE-9279
> URL: https://issues.apache.org/jira/browse/HIVE-9279
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>Priority: Blocker
> Fix For: encryption-branch
>
> Attachments: HIVE-9279.patch
>
>
> NO PRECOMMIT TESTS
> TestEncryptedHDFSCliDriver get exception "Could not execute test command" for
> encryption test cases.
> The stacktrace is:
> {quote}
> Unexpected exception java.lang.RuntimeException: Could not execute test
> command: Could not get CommandProcessor for command:
> at org.apache.hadoop.hive.ql.QTestUtil.executeTestCommand(QTestUtil.java:967)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:931)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:922)
> at
> org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.runTest(TestEncryptedHDFSCliDriver.java:150)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9279) TestEncryptedHDFSCliDriver get exception "Could not execute test command" for encryption test cases
[
https://issues.apache.org/jira/browse/HIVE-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Chen updated HIVE-9279:
Fix Version/s: encryption-branch
Status: Patch Available (was: Open)
> TestEncryptedHDFSCliDriver get exception "Could not execute test command" for
> encryption test cases
> ---
>
> Key: HIVE-9279
> URL: https://issues.apache.org/jira/browse/HIVE-9279
> Project: Hive
> Issue Type: Sub-task
>Reporter: Dong Chen
>Assignee: Dong Chen
>Priority: Blocker
> Fix For: encryption-branch
>
>
> NO PRECOMMIT TESTS
> TestEncryptedHDFSCliDriver get exception "Could not execute test command" for
> encryption test cases.
> The stacktrace is:
> {quote}
> Unexpected exception java.lang.RuntimeException: Could not execute test
> command: Could not get CommandProcessor for command:
> at org.apache.hadoop.hive.ql.QTestUtil.executeTestCommand(QTestUtil.java:967)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:931)
> at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:922)
> at
> org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.runTest(TestEncryptedHDFSCliDriver.java:150)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9279) TestEncryptedHDFSCliDriver get exception "Could not execute test command" for encryption test cases
Dong Chen created HIVE-9279:
---
Summary: TestEncryptedHDFSCliDriver get exception "Could not
execute test command" for encryption test cases
Key: HIVE-9279
URL: https://issues.apache.org/jira/browse/HIVE-9279
Project: Hive
Issue Type: Sub-task
Reporter: Dong Chen
Assignee: Dong Chen
Priority: Blocker
NO PRECOMMIT TESTS
TestEncryptedHDFSCliDriver get exception "Could not execute test command" for
encryption test cases.
The stacktrace is:
{quote}
Unexpected exception java.lang.RuntimeException: Could not execute test
command: Could not get CommandProcessor for command:
at org.apache.hadoop.hive.ql.QTestUtil.executeTestCommand(QTestUtil.java:967)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:931)
at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:922)
at
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.runTest(TestEncryptedHDFSCliDriver.java:150)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
