[jira] [Created] (HIVE-21716) alterPartitions does not take dbName like alterPartition
Jisu You created HIVE-21716: --- Summary: alterPartitions does not take dbName like alterPartition Key: HIVE-21716 URL: https://issues.apache.org/jira/browse/HIVE-21716 Project: Hive Issue Type: Improvement Components: Hive Reporter: Jisu You Assignee: Jisu You Fix For: 4.0.0 org.apache.hadoop.hive.ql.metadata.Hive::alterPartition org.apache.hadoop.hive.ql.metadata.Hive::alterPartitions Above two methods should only be different in one arg in my opinion. alterPartition should take in partition as one of the arg and alterPartitions should take List as one of the arg. However, there are quite a lot of differences between the two methods. This issue is created to tackle the difference between the two methods. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [VOTE] Apache Hive 2.3.5 Release Candidate 0
+1 For the source release: * check signatures of tarball * asked Alan to update the KEYS file with his current key * built code * ran a set of the unit tests .. Owen On Tue, May 7, 2019 at 4:57 PM Alan Gates wrote: > Apache Hive 2.3.5 Release Candidate 0 is available > here:http://people.apache.org/~gates/apache-hive-2.3.5-rc-0/ > > Maven artifacts are available here: > https://repository.apache.org/content/repositories/orgapachehive-1094/ > > Source tag for RC0 is release-2.3.5-rc0 > > Voting will conclude in 72 hours. > > Hive PMC Members: Please test and vote. > > Thanks. > > > Alan. >
Passing Hive Client Configuration to HiveMetaStore
Hi Team, I have a use case where I need to pass GCS bucket credentials to HiveMetaStore to get it authorized by StorageBasedAuthorizationProvider for GCS bucket. I had debugged and understood that Hive CLI's configuration (HiveConf -- populated from Hadoop Conf + hive-site.xml ) is not getting passed to HiveMetaStore. Is there any way we can send custom properties (HiveConf) to HiveMetaStore? P.S. : I could start HiveMetaStore with GCS credentials (by adding in core-site.xml) , but in my use case credentials supposed be send dynamically. Thank You, Shrikant Bang
[jira] [Created] (HIVE-21715) Adding a new partition (empty) partiton specified by location stops with exceptions
Zoltan Haindrich created HIVE-21715:
---
Summary: Adding a new partition (empty) partiton specified by
location stops with exceptions
Key: HIVE-21715
URL: https://issues.apache.org/jira/browse/HIVE-21715
Project: Hive
Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
{code}
create table supply (id int, part string, quantity int) partitioned by (day int)
stored as orc
location 'hdfs:///tmp/a1'
TBLPROPERTIES ('transactional'='true')
;
alter table supply add partition (day=20110103) location
'hdfs:///tmp/a3';
{code}
check exception:
{code}
org.apache.hadoop.hive.ql.metadata.HiveException: Wrong file format. Please
check the file's format.
at
org.apache.hadoop.hive.ql.exec.MoveTask.checkFileFormats(MoveTask.java:696)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:370)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:210)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
{code}
If the format check is disabled; an exception happens from AcidUtils; because
during checking it doesn't expect it to be empty.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-21714) INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting with ACID table
Ivan Suller created HIVE-21714: -- Summary: INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting with ACID table Key: HIVE-21714 URL: https://issues.apache.org/jira/browse/HIVE-21714 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Ivan Suller Assignee: Ivan Suller The issue of HIVE-18702 is present for ACID tables as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21713) Explain consistency Task/Operator
Zoltan Haindrich created HIVE-21713: --- Summary: Explain consistency Task/Operator Key: HIVE-21713 URL: https://issues.apache.org/jira/browse/HIVE-21713 Project: Hive Issue Type: Improvement Reporter: Zoltan Haindrich there seems to be some inconsistency what gets printed in the explain; for example "MoveTask" is shown as "Move Operator" in the explain Becase Tasks and Operators are basically different things - this should be corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 70474: HIVE-21407: Parquet predicate pushdown is not working correctly for char column types
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70474/#review215157 --- Ship it! Ship It! - Peter Vary On máj. 9, 2019, 7:51 de, Marta Kuczora wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/70474/ > --- > > (Updated máj. 9, 2019, 7:51 de) > > > Review request for hive and Peter Vary. > > > Bugs: HIVE-21407 > https://issues.apache.org/jira/browse/HIVE-21407 > > > Repository: hive-git > > > Description > --- > > The idea behind the patch is that for CHAR columns extend the predicate which > is pushed to Parquet with an “or” clause which contains the same expression > with a padded and a stripped value. > Example: > column c is a CHAR(10) type and the search expression is c='apple' > The predicate which is pushed to Parquet looked like c='apple ' before the > patch and it would look like (c='apple ' or c='apple') after the patch. > Since the value 'apple' is stored in Parquet without padding, the predicate > before the patch didn’t return any rows. With the patch it will return the > correct row. > Since on predicate level, there is no distinction between CHAR or VARCHAR, > the predicates for VARCHARs will be changed as well, so the result set > returned from Parquet will be wider than before. > Example: > A table contains a c VARCHAR(10) column and there is a row where c='apple' > and there is an other row where c='apple '. If the search expression is > c='apple ', both rows will be returned from Parquet after the patch. But > since Hive is doing an additional filtering on the rows returned from > Parquet, it won’t be a problem, the result set returned by Hive will contain > only the row with the value 'apple '. > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/io/parquet/LeafFilterFactory.java > be4c0d5 > > ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRecordReaderWrapper.java > 0210a0a > > ql/src/test/org/apache/hadoop/hive/ql/io/parquet/read/TestParquetFilterPredicate.java > d464046 > ql/src/test/queries/clientpositive/parquet_ppd_char.q 4230d8c > ql/src/test/queries/clientpositive/parquet_ppd_char2.q PRE-CREATION > ql/src/test/results/clientpositive/parquet_ppd_char2.q.out PRE-CREATION > > > Diff: https://reviews.apache.org/r/70474/diff/2/ > > > Testing > --- > > Added new q test for testing the PPD for char and varchar types. Also > extended the unit tests for the > ParquetFilterPredicateConverter.toFilterPredicate method. > > The TestParquetRecordReaderWrapper and the TestParquetFilterPredicate are > both testing the same thing, the behavior of the > ParquetFilterPredicateConverter.toFilterPredicate method. It doesn't make > sense to have tests for the same use case in different test classes, so moved > the test cases from the TestParquetRecordReaderWrapper to > TestParquetFilterPredicate. > > > Thanks, > > Marta Kuczora > >
Re: Review Request 70474: HIVE-21407: Parquet predicate pushdown is not working correctly for char column types
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70474/ --- (Updated May 9, 2019, 7:51 a.m.) Review request for hive and Peter Vary. Changes --- Fixed the whitespace issue. Bugs: HIVE-21407 https://issues.apache.org/jira/browse/HIVE-21407 Repository: hive-git Description --- The idea behind the patch is that for CHAR columns extend the predicate which is pushed to Parquet with an “or” clause which contains the same expression with a padded and a stripped value. Example: column c is a CHAR(10) type and the search expression is c='apple' The predicate which is pushed to Parquet looked like c='apple ' before the patch and it would look like (c='apple ' or c='apple') after the patch. Since the value 'apple' is stored in Parquet without padding, the predicate before the patch didn’t return any rows. With the patch it will return the correct row. Since on predicate level, there is no distinction between CHAR or VARCHAR, the predicates for VARCHARs will be changed as well, so the result set returned from Parquet will be wider than before. Example: A table contains a c VARCHAR(10) column and there is a row where c='apple' and there is an other row where c='apple '. If the search expression is c='apple ', both rows will be returned from Parquet after the patch. But since Hive is doing an additional filtering on the rows returned from Parquet, it won’t be a problem, the result set returned by Hive will contain only the row with the value 'apple '. Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/io/parquet/LeafFilterFactory.java be4c0d5 ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestParquetRecordReaderWrapper.java 0210a0a ql/src/test/org/apache/hadoop/hive/ql/io/parquet/read/TestParquetFilterPredicate.java d464046 ql/src/test/queries/clientpositive/parquet_ppd_char.q 4230d8c ql/src/test/queries/clientpositive/parquet_ppd_char2.q PRE-CREATION ql/src/test/results/clientpositive/parquet_ppd_char2.q.out PRE-CREATION Diff: https://reviews.apache.org/r/70474/diff/2/ Changes: https://reviews.apache.org/r/70474/diff/1-2/ Testing --- Added new q test for testing the PPD for char and varchar types. Also extended the unit tests for the ParquetFilterPredicateConverter.toFilterPredicate method. The TestParquetRecordReaderWrapper and the TestParquetFilterPredicate are both testing the same thing, the behavior of the ParquetFilterPredicateConverter.toFilterPredicate method. It doesn't make sense to have tests for the same use case in different test classes, so moved the test cases from the TestParquetRecordReaderWrapper to TestParquetFilterPredicate. Thanks, Marta Kuczora
