[jira] [Created] (HIVE-22816) QueryCache: Queries using views can have them cached after CTE expansion
Gopal Vijayaraghavan created HIVE-22816:
---
Summary: QueryCache: Queries using views can have them cached
after CTE expansion
Key: HIVE-22816
URL: https://issues.apache.org/jira/browse/HIVE-22816
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Gopal Vijayaraghavan
{code}
create view ss_null as select * from store_Sales where ss_Sold_date_sk is null;
select count(ss_ticket_number) from ss_null;
with ss_null_cte as
(select * from store_Sales where ss_Sold_date_sk is null)
select count(ss_ticket_number) from ss_null_cte;
{code}
Are treated differently by the query cache, however their execution is
identical.
CBO rewrites the view query into AST form as follows
{code}
SELECT COUNT(`ss_ticket_number`) AS `$f0`
FROM `tpcds_bin_partitioned_acid_orc_1`.`store_sales`
WHERE `ss_sold_date_sk` IS NULL
{code}
But retains the write-entity for the VIRTUAL_VIEW for Ranger authorization
{code}
0: jdbc:hive2://localhost:10013> explain dependency select count(distinct
ss_ticket_number) from ss_null;
++
| Explain |
++
|
{"input_tables":[{"tablename":"tpcds_bin_partitioned_acid_orc_1@ss_null","tabletype":"VIRTUAL_VIEW"},{"tablename":"tpcds_bin_partitioned_acid_orc_1@store_sales","tabletype":"MANAGED_TABLE","tableParents":"[tpcds_bin_partitioned_acid_orc_1@ss_null]"}],"input_partitions":[{"partitionName":"tpcds_bin_partitioned_acid_orc_1@store_sales@ss_sold_date_sk=__HIVE_DEFAULT_PARTITION__"}]}
|
++
{code}
Causing Query cache to print out
{code}
parse.CalcitePlanner: Not eligible for results caching - query contains
non-transactional tables [ss_null]
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22815) reduce the unnecessary file system object creation in MROutput
Richard Zhang created HIVE-22815: Summary: reduce the unnecessary file system object creation in MROutput Key: HIVE-22815 URL: https://issues.apache.org/jira/browse/HIVE-22815 Project: Hive Issue Type: Bug Reporter: Richard Zhang MROutput generates unnecessary file system object which may create long latency in Cloud environment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22814) ArrayIndexOutOfBound in the vectorization getDataTypePhysicalVariation
Ramesh Kumar Thangarajan created HIVE-22814: --- Summary: ArrayIndexOutOfBound in the vectorization getDataTypePhysicalVariation Key: HIVE-22814 URL: https://issues.apache.org/jira/browse/HIVE-22814 Project: Hive Issue Type: Bug Reporter: Ramesh Kumar Thangarajan Assignee: Ramesh Kumar Thangarajan ArrayIndexOutOfBound in the vectorization getDataTypePhysicalVariation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22813) Hive query fails if table location is in remote EZ and it's readonly
Purshotam Shah created HIVE-22813:
-
Summary: Hive query fails if table location is in remote EZ and
it's readonly
Key: HIVE-22813
URL: https://issues.apache.org/jira/browse/HIVE-22813
Project: Hive
Issue Type: Bug
Reporter: Purshotam Shah
{code}
[purushah@gwrd352n21 ~]$ hive
hive> select * from puru_db.page_view_ez;
FAILED: SemanticException Unable to compare key strength for
hdfs://nn1/<>/puru_db_ez/page_view_ez and
hdfs://nn2:8020/tmp/puru/d558ac89-1359-424c-92ee-d0fefa8e6593/hive_2020-01-31_19-46-55_114_644945433042922-1/-mr-1
: java.lang.IllegalArgumentException: Wrong FS:
hdfs://nn1:8020/<>/puru_db_ez/page_view_ez, expected: hdfs://nn2
hive>
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Review Request 72074: HIVE-21215: Read Parquet INT64 timestamp
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72074/ --- Review request for hive, Karen Coppage and Peter Vary. Bugs: HIVE-21215 https://issues.apache.org/jira/browse/HIVE-21215 Repository: hive-git Description --- Implemented the read path for Parquet INT64 timestamp. Diffs - common/src/java/org/apache/hadoop/hive/common/type/Timestamp.java f2c1493f56 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/ETypeConverter.java d67b030648 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/timestamp/ParquetTimestampUtils.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/ParquetDataColumnReaderFactory.java 519bd813e9 ql/src/java/org/apache/hadoop/hive/ql/io/parquet/vector/VectorizedPrimitiveColumnReader.java 2803baf90c ql/src/test/org/apache/hadoop/hive/ql/io/parquet/convert/TestETypeConverter.java f6ee57140c Diff: https://reviews.apache.org/r/72074/diff/1/ Testing --- Thanks, Marta Kuczora
Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71904/ --- (Updated Jan. 31, 2020, 4:12 p.m.) Review request for hive, Gopal V and Peter Vary. Bugs: HIVE-21164 https://issues.apache.org/jira/browse/HIVE-21164 Repository: hive-git Description --- Extended the original patch with saving the task attempt ids in the file names and also fixed some bugs in the original patch. With this fix, inserting into an ACID table would not use move task to place the generated files into the final directory. It will inserts every files to the final directory and then clean up the files which are not needed (like written by failed task attempts). Also fixed the replication tests which failed for the original patch as well. Diffs (updated) - hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java da677c7977 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 056cd27496 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java 31d15fdef9 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorTestUtil.java c2aa73b5f1 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java 4c0137 ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 9a3258115b ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 8980a6292a ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java c4c56f8477 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java b8a0f0465c ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 398698ec06 ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java 2543dc6fc4 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 73ca658d9c ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5fcc367cc9 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java c102a69f8f ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java bb70db4524 ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java af14e628b3 ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 2c4b69b2fe ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 48e9afc496 ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java cfd7290762 ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 70ae85c458 ql/src/test/results/clientpositive/acid_subquery.q.out 1dc1775557 ql/src/test/results/clientpositive/create_transactional_full_acid.q.out e324d5ec43 ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out 61b0057adb ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out fbf4e481f1 ql/src/test/results/clientpositive/llap/insert_overwrite.q.out fbc3326b39 ql/src/test/results/clientpositive/llap/mm_all.q.out 226f2a9374 ql/src/test/results/clientpositive/mm_all.q.out 143ebd69f9 streaming/src/test/org/apache/hive/streaming/TestStreaming.java 35a220facd Diff: https://reviews.apache.org/r/71904/diff/3/ Changes: https://reviews.apache.org/r/71904/diff/2-3/ Testing ---
[jira] [Created] (HIVE-22812) FilterOperator may encounter ClassCastExceptions for multi-insert queries
Zoltan Haindrich created HIVE-22812:
---
Summary: FilterOperator may encounter ClassCastExceptions for
multi-insert queries
Key: HIVE-22812
URL: https://issues.apache.org/jira/browse/HIVE-22812
Project: Hive
Issue Type: Bug
Reporter: Zoltan Haindrich
after initialization
{code}
drop table if exists t;
drop table if exists s;
drop table if exists t_offset;
create table t(id integer, cnt integer);
create table s(id integer, cnt integer);
create table t_offset(offset integer);
insert into t_offset values(0);
insert into s values(1,1);
{code}
the query:
{code:java}
from (select * from s) s1
insert overwrite table t_offset select max(s1.id) group by 1
insert into t select id,cnt where id
;
{code}
results in
{code}
], TaskAttempt 3 failed, info=[Error: Error while running task ( failure ) :
attempt_1580482492339_0019_5_00_00_3:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:403)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
... 19 more
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Boolean
at
org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:124)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22811) Statistics are not exploit in nested cases
Zoltan Haindrich created HIVE-22811:
---
Summary: Statistics are not exploit in nested cases
Key: HIVE-22811
URL: https://issues.apache.org/jira/browse/HIVE-22811
Project: Hive
Issue Type: Improvement
Components: Statistics
Reporter: Zoltan Haindrich
The statsOptimizer is able to use min/max/etc values to service simple queries
{code}
(select max(id) from t t0)
{code}
however the same doesn't happen for queries like:
{code}
explain select * from u where u.id>(select max(id) from t t0);
{code}
explain:
{code}
| Plan optimized by CBO. |
||
| Vertex dependency in root stage|
| Reducer 3 <- Map 1 (BROADCAST_EDGE), Map 2 (CUSTOM_SIMPLE_EDGE) |
||
| Stage-0|
| Fetch Operator |
| limit:-1 |
| Stage-1|
| Reducer 3 vectorized |
| File Output Operator [FS_31] |
| Select Operator [SEL_30] (rows=1 width=8) |
| Output:["_col0","_col1"] |
| Filter Operator [FIL_29] (rows=1 width=12) |
| predicate:(_col0 > _col2) |
| Map Join Operator [MAPJOIN_28] (rows=3 width=12) |
| Conds:(Inner),Output:["_col0","_col1","_col2"] |
| <-Map 1 [BROADCAST_EDGE] vectorized|
| BROADCAST [RS_25]|
| Select Operator [SEL_24] (rows=3 width=8) |
| Output:["_col0","_col1"] |
| Filter Operator [FIL_23] (rows=3 width=8) |
| predicate:id is not null |
| TableScan [TS_0] (rows=3 width=8) |
|
default@u,u,Tbl:COMPLETE,Col:COMPLETE,Output:["id","cnt"] |
| <-Filter Operator [FIL_27] (rows=1 width=4) |
| predicate:_col0 is not null|
| Group By Operator [GBY_26] (rows=1 width=4) |
| Output:["_col0"],aggregations:["max(VALUE._col0)"] |
| <-Map 2 [CUSTOM_SIMPLE_EDGE] vectorized |
| PARTITION_ONLY_SHUFFLE [RS_22] |
| Group By Operator [GBY_21] (rows=1 width=4) |
| Output:["_col0"],aggregations:["max(id)"] |
| Select Operator [SEL_20] (rows=4 width=4) |
| Output:["id"] |
| TableScan [TS_3] (rows=4 width=4) |
|
default@t,t0,Tbl:COMPLETE,Col:COMPLETE,Output:["id"] |
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Review Request 72073: HIVE-22808
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72073/ --- Review request for hive, Jesús Camacho Rodríguez and Vineet Garg. Bugs: HIVE-22808 https://issues.apache.org/jira/browse/HIVE-22808 Repository: hive-git Description --- HiveRelFieldTrimmer does not handle HiveTableFunctionScan Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveTableFunctionScan.java ffa2a1f262 ql/src/test/results/clientpositive/intersect_all_rj.q.out b8ff98ae79 ql/src/test/results/clientpositive/llap/intersect_all_rj.q.out cdfbc2239e Diff: https://reviews.apache.org/r/72073/diff/1/ Testing --- mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestCliDriver -Dqfile=intersect_all_rj.q -pl itests/qtest -Pitests Thanks, Krisztian Kasa
[jira] [Created] (HIVE-22810) Add drop scheduled query if exists support
Zoltan Haindrich created HIVE-22810: --- Summary: Add drop scheduled query if exists support Key: HIVE-22810 URL: https://issues.apache.org/jira/browse/HIVE-22810 Project: Hive Issue Type: Sub-task Reporter: Zoltan Haindrich Could make life easier. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22809) Support materialized view rebuild as a scheduled query
Zoltan Haindrich created HIVE-22809: --- Summary: Support materialized view rebuild as a scheduled query Key: HIVE-22809 URL: https://issues.apache.org/jira/browse/HIVE-22809 Project: Hive Issue Type: Sub-task Components: Materialized views Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich scheduled queries build heavily on unparsing the original query - to free it from context free stuff; meanwhile the mv rebuild replaces the query with a new command -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22808) HiveRelFieldTrimmer does not handle HiveTableFunctionScan
Krisztian Kasa created HIVE-22808:
-
Summary: HiveRelFieldTrimmer does not handle HiveTableFunctionScan
Key: HIVE-22808
URL: https://issues.apache.org/jira/browse/HIVE-22808
Project: Hive
Issue Type: Bug
Components: Query Planning
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa
*Repro*
{code:java}
CREATE TABLE table_16 (
timestamp_col_19timestamp,
timestamp_col_29timestamp,
int_col_27 int,
int_col_39 int,
boolean_col_18 boolean,
varchar0045_col_23 varchar(45)
);
CREATE TABLE table_7 (
int_col_10 int,
bigint_col_3bigint
);
CREATE TABLE table_10 (
boolean_col_8 boolean,
boolean_col_16 boolean,
timestamp_col_5 timestamp,
timestamp_col_15timestamp,
timestamp_col_30timestamp,
decimal3825_col_26 decimal(38, 25),
smallint_col_9 smallint,
int_col_18 int
);
explain cbo
SELECT
DISTINCT COALESCE(a4.timestamp_col_15, IF(a4.boolean_col_16,
a4.timestamp_col_30, a4.timestamp_col_5)) AS timestamp_col
FROM table_7 a3
RIGHT JOIN table_10 a4
WHERE (a3.bigint_col_3) >= (a4.int_col_18)
INTERSECT ALL
SELECT
COALESCE(LEAST(
COALESCE(a1.timestamp_col_19, CAST('2010-03-29 00:00:00' AS TIMESTAMP)),
COALESCE(a1.timestamp_col_29, CAST('2014-08-16 00:00:00' AS TIMESTAMP))
),
GREATEST(COALESCE(a1.timestamp_col_19, CAST('2013-07-01 00:00:00' AS
TIMESTAMP)),
COALESCE(a1.timestamp_col_29, CAST('2028-06-18 00:00:00' AS TIMESTAMP)))
) AS timestamp_col
FROM table_16 a1
GROUP BY COALESCE(LEAST(
COALESCE(a1.timestamp_col_19, CAST('2010-03-29 00:00:00' AS TIMESTAMP)),
COALESCE(a1.timestamp_col_29, CAST('2014-08-16 00:00:00' AS TIMESTAMP))
),
GREATEST(
COALESCE(a1.timestamp_col_19, CAST('2013-07-01 00:00:00' AS TIMESTAMP)),
COALESCE(a1.timestamp_col_29, CAST('2028-06-18 00:00:00' AS TIMESTAMP)))
);
{code}
CBO Plan contains unnecessary columns or all columns from a table in
projections like:
{code:java}
HiveProject(int_col_10=[$0], bigint_col_3=[$1],
BLOCK__OFFSET__INSIDE__FILE=[$2], INPUT__FILE__NAME=[$3],
CAST=[CAST($4):RecordType(BIGINT writeid, INTEGER bucketid, BIGINT rowid)])
{code}
*Cause*
The plan contains a HiveTableFunctionScan operator:
{code:java}
HiveTableFunctionScan(invocation=[replicate_rows($0, $1)],
rowType=[RecordType(BIGINT $f0, TIMESTAMP(9) $f1)])
{code}
HiveTableFunctionScan is not handled by HiveRelFieldTrimmer nor RelFieldTrimmer
which suppose to remove unused columns in the
CalcitePlanner.applyPreJoinOrderingTransforms(...) phase. The whole subtree
rooted from HiveTableFunctionScan is ignored.
Whole plan:
{code:java}
CBO PLAN:
HiveProject($f0=[$1])
HiveTableFunctionScan(invocation=[replicate_rows($0, $1)],
rowType=[RecordType(BIGINT $f0, TIMESTAMP(9) $f1)])
HiveProject($f0=[$2], $f1=[$0])
HiveFilter(condition=[=($1, 2)])
HiveAggregate(group=[{0}], agg#0=[count($1)], agg#1=[min($1)])
HiveProject($f0=[$0], $f1=[$1])
HiveUnion(all=[true])
HiveProject($f0=[$0], $f1=[$1])
HiveAggregate(group=[{0}], agg#0=[count()])
HiveProject($f0=[$0])
HiveAggregate(group=[{0}])
HiveProject($f0=[CASE(IS NOT NULL($7), $7, if($5, $8,
$6))])
HiveJoin(condition=[>=($1, $13)], joinType=[inner],
algorithm=[none], cost=[not available])
HiveProject(int_col_10=[$0], bigint_col_3=[$1],
BLOCK__OFFSET__INSIDE__FILE=[$2], INPUT__FILE__NAME=[$3],
CAST=[CAST($4):RecordType(BIGINT writeid, INTEGER bucketid, BIGINT rowid)])
HiveFilter(condition=[IS NOT NULL($1)])
HiveTableScan(table=[[default, table_7]],
table:alias=[a3])
HiveProject(boolean_col_16=[$0],
timestamp_col_5=[$1], timestamp_col_15=[$2], timestamp_col_30=[$3],
int_col_18=[$4], BLOCK__OFFSET__INSIDE__FILE=[$5], INPUT__FILE__NAME=[$6],
ROW__ID=[$7], CAST=[CAST($4):BIGINT])
HiveFilter(condition=[IS NOT NULL(CAST($4):BIGINT)])
HiveTableScan(table=[[default, table_10]],
table:alias=[a4])
HiveProject($f0=[$0], $f1=[$1])
HiveAggregate(group=[{0}], agg#0=[count()])
HiveProject($f0=[$0])
HiveAggregate(group=[{0}])
HiveProject($f0=[CASE(IS NOT NULL(least(CASE(IS NOT
NULL($0), $0, 2010-03-29 00:00:00:TIMESTAMP(9)), CASE(IS NOT NULL($1), $1,
2014-08-16 00:00:00:TIMESTAMP(9, least(CASE(IS NOT NULL($0), $0, 2010-03-29
00:00:00:TIMESTAMP(9)), CASE(IS NOT NULL($1), $1, 2014-08-16
00:00:00:TIMESTAMP(9))), greatest(CASE(IS NOT NULL($0), $0, 2013-07-01
00:00:00:TIMESTAMP(9)), CASE(IS NOT NULL($1), $1, 2028-06-18
00:00:00:TIMESTAMP(
[jira] [Created] (HIVE-22807) NoSuchMethodError when running hive or schematool
Reinhard Eilmsteiner created HIVE-22807: --- Summary: NoSuchMethodError when running hive or schematool Key: HIVE-22807 URL: https://issues.apache.org/jira/browse/HIVE-22807 Project: Hive Issue Type: Bug Affects Versions: 3.1.2 Environment: jdk1.8.0_241 hadoop 3.2.1 Reporter: Reinhard Eilmsteiner After execution of the steps described in [https://cwiki.apache.org/confluence/display/Hive/GettingStarted?show-miniview |https://cwiki.apache.org/confluence/display/Hive/GettingStarted?show-miniview]I run into the following exception when executing hive and schematool: Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536) at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554) at org.apache.hadoop.mapred.JobConf.(JobConf.java:448) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141) at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:5104) at org.apache.hive.beeline.HiveSchemaTool.(HiveSchemaTool.java:96) at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22806) Missing materialized view rewrite in case the filter is further narrowed
Zoltan Haindrich created HIVE-22806:
---
Summary: Missing materialized view rewrite in case the filter is
further narrowed
Key: HIVE-22806
URL: https://issues.apache.org/jira/browse/HIVE-22806
Project: Hive
Issue Type: Bug
Components: Materialized views
Reporter: Zoltan Haindrich
I was checking some basic things when I've noticed that mv rewriting doesn't
kick in for some cases:
{code}
explain
SELECT empid, deptname
FROM emps
JOIN depts
using (deptno)
WHERE hire_date >= 600
AND hire_date <= 1200-- depending on the presence of this condition the
rewrite may not happen
;
{code}
qtest:
{code}
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
set hive.strict.checks.cartesian.product=false;
set hive.stats.fetch.column.stats=true;
set hive.materializedview.rewriting=true;
-- create some tables
CREATE TABLE emps (
empid INT,
deptno INT,
name VARCHAR(256),
salary FLOAT,
hire_date int)
STORED AS ORC
TBLPROPERTIES ('transactional'='true');
CREATE TABLE depts (
deptno INT,
deptname VARCHAR(256),
locationid INT)
STORED AS ORC
TBLPROPERTIES ('transactional'='true');
-- load data
insert into emps values (100, 10, 'Bill', 1, 1000), (200, 20, 'Eric', 8000,
500),
(150, 10, 'Sebastian', 7000, null), (110, 10, 'Theodore', 1, 250), (120,
10, 'Bill', 1, 250)
;
insert into depts values (10, 'Sales', 10), (30, 'Marketing', null), (20, 'HR',
20);
alter table emps add constraint pk1 primary key (empid) disable novalidate rely;
alter table depts add constraint pk2 primary key (deptno) disable novalidate
rely;
alter table emps add constraint fk1 foreign key (deptno) references
depts(deptno) disable novalidate rely;
-- create mv
CREATE MATERIALIZED VIEW mv1
AS
SELECT empid, deptname, hire_date
FROM emps JOIN depts
using (deptno)
-- ON (emps.deptno = depts.deptno)
WHERE hire_date >= 500;
-- expected to see that materialzed view is being used; however it doesnt:
explain
SELECT empid, deptname
FROM emps
JOIN depts
using (deptno)
WHERE hire_date >= 600
AND hire_date <= 1200
;
-- now we can see that the materialzed view is being used:
explain
SELECT empid, deptname
FROM emps
JOIN depts
using (deptno)
WHERE hire_date >= 600
--AND hire_date <= 1200
;
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22805) Vectorization with conditional array or map is not implemented and throws an error
Peter Vary created HIVE-22805:
-
Summary: Vectorization with conditional array or map is not
implemented and throws an error
Key: HIVE-22805
URL: https://issues.apache.org/jira/browse/HIVE-22805
Project: Hive
Issue Type: Bug
Components: Vectorization
Reporter: Peter Vary
Assignee: Peter Vary
The following queries throw an "Not supported" Exception:
{code}
DROP TABLE IF EXISTS cond_vector;
CREATE TABLE cond_vector(a STRING) STORED AS ORC;
INSERT OVERWRITE TABLE cond_vector VALUES("a/b");
set hive.fetch.task.conversion=minimal;
set hive.execution.mode=container;
SELECT IF(1=1, MAP("a","b"), NULL) FROM cond_vector;
EXPLAIN VECTORIZATION DETAIL SELECT IF(1=1, MAP("Mathematics","78"), NULL) FROM
cond_vector;
SELECT IF(1=1, ARRAY("c", "d"), NULL) FROM cond_vector;
EXPLAIN VECTORIZATION DETAIL SELECT IF(1=1, ARRAY("a", "b"), NULL) FROM
cond_vector;
{code}
One example of the Exception (maybe not exactly this on 4.0.0 branch, but close
enough to check):
{code}
Caused by: java.lang.RuntimeException: Not supported
at
org.apache.hadoop.hive.ql.exec.vector.ListColumnVector.copySelected(ListColumnVector.java:161)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprCondExprNull.evaluate(IfExprCondExprNull.java:87)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprCondExprCondExpr.evaluate(IfExprCondExprCondExpr.java:95)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprCondExprBase.conditionalEvaluate(IfExprCondExprBase.java:68)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprCondExprCondExpr.evaluate(IfExprCondExprCondExpr.java:110)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprCondExprCondExpr.evaluate(IfExprCondExprCondExpr.java:95)
at
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:146)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22804) Ensure ANSI quotes are used for mysql connections
Zoltan Chovan created HIVE-22804:
Summary: Ensure ANSI quotes are used for mysql connections
Key: HIVE-22804
URL: https://issues.apache.org/jira/browse/HIVE-22804
Project: Hive
Issue Type: Improvement
Reporter: Zoltan Chovan
Assignee: Zoltan Chovan
Recent changes in direct sql queries to resolve postgres issues(e.g. TxnHandler
in HIVE-22663 ) break compatibility with mysql backend db.
A workaround for these issues is to add a session config to the mysql
connection string, e.g.:
{code:java}
jdbc:mysql://localhost:3306/db?sessionVariables=sql_mode=ANSI_QUOTES
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 72028: HIVE-22729: Provide a failure reason for failed compactions
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72028/#review219450 --- Ship it! - Peter Vary On jan. 30, 2020, 10:23 de, Laszlo Pinter wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72028/ > --- > > (Updated jan. 30, 2020, 10:23 de) > > > Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary. > > > Repository: hive-git > > > Description > --- > > HIVE-22729: Provide a failure reason for failed compactions > > > Diffs > - > > metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql > 5421d4d8141becae4e0de6a039bf7c46f0b109bb > metastore/scripts/upgrade/hive/upgrade-3.1.0-to-4.0.0.hive.sql > 041190653898a39ef96c6c2bf71c4f4485f6a1a5 > > ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsDesc.java > 9348efc5a12b50f55f5952094882e941158405fd > > ql/src/java/org/apache/hadoop/hive/ql/ddl/process/show/compactions/ShowCompactionsOperation.java > 517d88237cc3b8f0316727bf1eebfc6535152fae > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java > 6f642901203ab73699ed694009d48ca77263fb10 > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java > dedc990d0f1e9123497f0fb7c7b9945c7b29bde2 > ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java > 5aff71e0e981c429f85663300d3e5c21089529a9 > > ql/src/test/org/apache/hadoop/hive/metastore/txn/TestCompactionTxnHandler.java > e5895547e6006f30a37b5ba0b1ce42253129d3b6 > ql/src/test/results/clientpositive/dbtxnmgr_showlocks.q.out > 03c6724ec2e50ae1f7c642339c1806d0786a9ec5 > > standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java > 4aee45ce5f0e534823194bc84d13b88210ce0b3c > > standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ShowCompactResponseElement.java > 8a5682a013b24f8dcf7ad3fdb0b0b606d82cc7c0 > > standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php > e8556dcea68f34336df2925c4108e71185d6377f > > standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py > b05e61e84a310273911ef592258bcd3b34e87734 > > standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb > 868cf69200f69aa89e82b34e22ee0ad792e6d025 > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift > 61a94fee4d82a714c12aeeb27f31e24774592c98 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java > ba45f3945274853fdc84487d93c4c00ff2982541 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java > aded6f5486cc840f397347b39049310009fd3bad > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java > da5dd61d08e2ca8fe5e80ffdf9fb4a6f4c4d0ba3 > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java > c2c97d96c6cc98f9746069fa725d17d12f6c8642 > > standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql > 67102718867233f29ddb2ea8ec3fbcb6560c6c30 > > standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql > ae0a32541a4bb9179b2bb71ae9f9098d7b35a88e > > standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql > 221d4f1fffb682aaec3af22a339e7a3077a75f6a > > standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql > bc98d5fc4a5637988c97f2e5a0e02d3be16ae0cb > > standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql > dd761a66db4826580a67d64879e4c85278b8e20c > > standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql > 6a040a6a64c2086b5eb68a397697c9e2d2ca4d76 > > standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql > f5ec1ba1aff89d02b66d6a2cd1da8de1b3b08d06 > > standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql > c7738be2732b839aa2b460733c092e368909f935 > > standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql > 455f98b72578ff977e29301cd2fc595ae80ee4ca > > standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql > 5c39b0d9f4d27ab82ef44392818c1810cb7664ce > > > Diff: https://reviews.apache.org/r/72028/diff/3/ > > > Testing > --- > > > Thanks, > > Laszlo Pinter > >
[jira] [Created] (HIVE-22803) Mark scheduled queries executions to help end-user identify it easier
Zoltan Haindrich created HIVE-22803:
---
Summary: Mark scheduled queries executions to help end-user
identify it easier
Key: HIVE-22803
URL: https://issues.apache.org/jira/browse/HIVE-22803
Project: Hive
Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
scheduled queries are executed as-is;
it might probably help a lot in field deployments to have a hint where that
query is coming from; I'm thinking of prefixing the query with some comment:
{code}
/*schedule:sc1*/ select 1
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22802) Scheduled query related views should restrict access
Zoltan Haindrich created HIVE-22802: --- Summary: Scheduled query related views should restrict access Key: HIVE-22802 URL: https://issues.apache.org/jira/browse/HIVE-22802 Project: Hive Issue Type: Sub-task Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich * only show schedules from the actual namespace * only show schedules for the actual user unless admin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22801) Debug log is flooded with some debug dump stack
Zoltan Haindrich created HIVE-22801:
---
Summary: Debug log is flooded with some debug dump stack
Key: HIVE-22801
URL: https://issues.apache.org/jira/browse/HIVE-22801
Project: Hive
Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich
it makes the debug log more or less useless
{code}
2020-01-31T12:18:39,441 DEBUG [NotificationEventPoll 0] metastore.ObjectStore:
Commit transaction: count = 0, isactive true
java.lang.Exception: Debug Dump Stack Trace (Not an Exception)
at
org.apache.hadoop.hive.metastore.ObjectStore.debugLog(ObjectStore.java:9685)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:475)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.ObjectStore.getNextNotification(ObjectStore.java:9956)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy37.getNextNotification(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_next_notification(HiveMetaStore.java:8485)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy39.get_next_notification(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getNextNotification(HiveMetaStoreClient.java:3391)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy40.getNextNotification(Unknown Source) ~[?:?]
at
org.apache.hadoop.hive.ql.metadata.events.EventUtils$MSClientNotificationFetcher.getNextNotificationEvents(EventUtils.java:107)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.fetchNextBatch(EventUtils.java:159)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.events.EventUtils$NotificationEventIterator.hasNext(EventUtils.java:189)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.events.NotificationEventPoll$Poller.run(NotificationEventPoll.java:139)
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_222]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
~[?:1.8.0_222]
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Review Request 71904: HIVE-21164: ACID: explore how we can avoid a move step during inserts/compaction
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71904/ --- (Updated Jan. 31, 2020, 10:17 a.m.) Review request for hive, Gopal V and Peter Vary. Changes --- Rebased the patch Bugs: HIVE-21164 https://issues.apache.org/jira/browse/HIVE-21164 Repository: hive-git Description --- Extended the original patch with saving the task attempt ids in the file names and also fixed some bugs in the original patch. With this fix, inserting into an ACID table would not use move task to place the generated files into the final directory. It will inserts every files to the final directory and then clean up the files which are not needed (like written by failed task attempts). Also fixed the replication tests which failed for the original patch as well. Diffs (updated) - hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java da677c7977 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java 056cd27496 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java 31d15fdef9 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCrudCompactorOnTez.java 4c0137 ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 9a3258115b ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 9ad4e71482 ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 06e4ebee82 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6c67bc7dd8 ql/src/java/org/apache/hadoop/hive/ql/io/AcidInputFormat.java bba3960102 ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java 1e8bb223f2 ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 2f5ec5270c ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 8980a6292a ql/src/java/org/apache/hadoop/hive/ql/io/RecordUpdater.java 737e6774b7 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 76984abd0a ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat.java c4c56f8477 ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java b8a0f0465c ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 398698ec06 ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java 2543dc6fc4 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 7f061d4a6b ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 73ca658d9c ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 5fcc367cc9 ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java c102a69f8f ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java ecc7bdee4d ql/src/java/org/apache/hadoop/hive/ql/plan/LoadTableDesc.java bed05819b5 ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java bb70db4524 ql/src/java/org/apache/hadoop/hive/ql/util/UpgradeTool.java 58e6289583 ql/src/test/org/apache/hadoop/hive/ql/TestTxnAddPartition.java c9cb6692df ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 842140815d ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 88ca683173 ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 908ceb43fc ql/src/test/org/apache/hadoop/hive/ql/TestTxnConcatenate.java 8676e0db11 ql/src/test/org/apache/hadoop/hive/ql/TestTxnExIm.java 66b2b2768b ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java bb55d9fd79 ql/src/test/org/apache/hadoop/hive/ql/TestTxnNoBuckets.java ea6b1d9bec ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java af14e628b3 ql/src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java 83db48e758 ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java 2c4b69b2fe ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 48e9afc496 ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java cfd7290762 ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestWorker.java 70ae85c458 ql/src/test/results/clientpositive/acid_subquery.q.out 1dc1775557 ql/src/test/results/clientpositive/create_transactional_full_acid.q.out e324d5ec43 ql/src/test/results/clientpositive/encrypted/encryption_insert_partition_dynamic.q.out 61b0057adb ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out fbf4e481f1 ql/src/test/results/clientpositive/llap/insert_overwrite.q.out fbc3326b39 ql/src/test/results/clientpositive/llap/mm_all.q.out 226f2a9374 ql/src/test/results/clientpositive/mm_all.q.out 143ebd69f9 streaming/src/test/org/apache/hive/streaming/TestStreaming.java 35a220facd Diff: https://reviews.apache.org/r/71904/diff/2/ Changes: https://reviews.apache.org/r/71904/diff/1-2/ Testing --- Had to modify some tests because of the file name changes. Also added
Re: Review Request 72043: HIVE-21487: COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72043/#review219380 --- Have you tested the sqls on every db? If they run correctly on all supported DB, then +1 from my side Thanks, Peter - Peter Vary On jan. 24, 2020, 10 de, Laszlo Pinter wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72043/ > --- > > (Updated jan. 24, 2020, 10 de) > > > Review request for hive, Denys Kuzmenko, Karen Coppage, and Peter Vary. > > > Repository: hive-git > > > Description > --- > > HIVE-21487: COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing > appropriate indexes > > > Diffs > - > > > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnDbUtil.java > da5dd61d08e2ca8fe5e80ffdf9fb4a6f4c4d0ba3 > > standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql > 67102718867233f29ddb2ea8ec3fbcb6560c6c30 > > standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql > ae0a32541a4bb9179b2bb71ae9f9098d7b35a88e > > standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql > 221d4f1fffb682aaec3af22a339e7a3077a75f6a > > standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql > bc98d5fc4a5637988c97f2e5a0e02d3be16ae0cb > > standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql > dd761a66db4826580a67d64879e4c85278b8e20c > > standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql > 6a040a6a64c2086b5eb68a397697c9e2d2ca4d76 > > standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql > f5ec1ba1aff89d02b66d6a2cd1da8de1b3b08d06 > > standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql > c7738be2732b839aa2b460733c092e368909f935 > > standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql > 455f98b72578ff977e29301cd2fc595ae80ee4ca > > standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql > 5c39b0d9f4d27ab82ef44392818c1810cb7664ce > > > Diff: https://reviews.apache.org/r/72043/diff/1/ > > > Testing > --- > > > Thanks, > > Laszlo Pinter > >
[jira] [Created] (HIVE-22800) GenericUDFOPDTIPlus should support varchar
Rajesh Balamohan created HIVE-22800:
---
Summary: GenericUDFOPDTIPlus should support varchar
Key: HIVE-22800
URL: https://issues.apache.org/jira/browse/HIVE-22800
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: Rajesh Balamohan
{noformat}
create table test(d_date varchar(10) stored as orc;
select d_date + INTERVAL(5) DAY from test;
Error: Error while compiling statement: FAILED: SemanticException [Error
10014]: Line 1:7 Wrong arguments '5': No matching method for class
org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPDTIPlus with (varchar(10),
interval_day_time) (state=42000,code=10014)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22799) HiveMetaStoreAuthorizer parses conf on every HMS invocation
Rajesh Balamohan created HIVE-22799:
---
Summary: HiveMetaStoreAuthorizer parses conf on every HMS
invocation
Key: HIVE-22799
URL: https://issues.apache.org/jira/browse/HIVE-22799
Project: Hive
Issue Type: Improvement
Components: Metastore
Reporter: Rajesh Balamohan
Stacktrace may not match exactly master branch. But in master as well, it is
not very different.
{noformat}
at org.apache.hadoop.util.StringInterner.weakIntern(StringInterner.java:71)
at
org.apache.hadoop.conf.Configuration$Parser.handleEndElement(Configuration.java:3273)
at
org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3354)
at
org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3137)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3030)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2991)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2871)
- locked <0x0005cbe60ad0> (a org.apache.hadoop.mapred.JobConf)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1389)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1361)
at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:518)
at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:536)
at org.apache.hadoop.mapred.JobConf.(JobConf.java:430)
at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5482)
at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:5450)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.createHiveMetaStoreAuthorizer(HiveMetaStoreAuthorizer.java:450)
at
org.apache.hadoop.hive.ql.security.authorization.plugin.metastore.HiveMetaStoreAuthorizer.onEvent(HiveMetaStoreAuthorizer.java:100)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.firePreEvent(HiveMetaStore.java:3835)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_database_req(HiveMetaStore.java:1655)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy28.get_database_req(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_database_req.getResult(ThriftHiveMetastore.java:15671)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$get_database_req.getResult(ThriftHiveMetastore.java:15655)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22798) Fix/Optimize: PrimitiveTypeInfo::getPrimitiveTypeEntry
Rajesh Balamohan created HIVE-22798: --- Summary: Fix/Optimize: PrimitiveTypeInfo::getPrimitiveTypeEntry Key: HIVE-22798 URL: https://issues.apache.org/jira/browse/HIVE-22798 Project: Hive Issue Type: Improvement Components: Hive Reporter: Rajesh Balamohan Attachments: image-2020-01-31-14-22-45-372.png !image-2020-01-31-14-22-45-372.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
