java.lang.IllegalAccessError:tried to access method com.goole.common.collect.Iterators.empty()Lcom/google/commmon/collect/UnmodifiableIterator;from class org.apache.hadoop.hive.ql.exec.FetchOperator

[qq]

Hello:
      The version of hive is 2.3.3, and the version of Hadoop is 
3.2.1. 
When I execute show tables on the command line of hive beeline, the following 
error occurs:
Error message in picture
How can I solve it?
     
 thinks.
 I am looking forward to your reply??
 

Re: [VOTE] Should we release Hive Storage API 2.7.2-rc1?

[Gopal V]

Hi,

Validated checksums, signatures, built and verified against latest orc.

Since I was lazy enough to automate this, here's a script for others who 
might not have voted (or want to add things to this).


https://github.com/t3rmin4t0r/verify-asf-releases + make -f 
Makefile.storage-api


should do all the checks I did, unattended.

Cheers,
Gopal

On 5/5/20 7:49 AM, Jesus Camacho Rodriguez wrote:

All,

I'd like to make a storage-api release with HIVE-22959
 and HIVE-23215 <
https://issues.apache.org/jira/browse/HIVE-23215> in it.

Should we release the following artifacts as Hive Storage API 2.7.2?

tar: http://home.apache.org/~jcamacho/hive-storage-2.7.2/
tag: https://github.com/apache/hive/releases/tag/storage-release-2.7.2-rc1
jiras: https://issues.apache.org/jira/projects/HIVE/versions/12347828

Thanks!

-Jesús

Is Hive accepting patches for 2.3.x?

[Shashank Pedamallu]

Hi all,

I'm Shashank and I work as a software engineer in the Data Platform team at
Lyft. We are using the 2.3.6 version of Hive. We have made some changes in
the Hive codebase (improve build time by adding parallelism to unit-tests,
etc). I wanted to check if the open-source community is accepting patches
for 2.3.x version of Hive. I do know Hive-2.3.7 was released less than a
month but would like to know if these new patches are specifically around
bug-fixes or can also be features, code-cleanups, improvements, etc.

Thanks,
Shashank

Shashank Pedamallu
Software Engineer, Analytics Infrastructure
(610)-504-1725

[jira] [Created] (HIVE-23425) HS2 disconnect should not affect DDL operation status

[Sam An (Jira)]

Sam An created HIVE-23425:
-

 Summary: HS2 disconnect should not affect DDL operation status
 Key: HIVE-23425
 URL: https://issues.apache.org/jira/browse/HIVE-23425
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Sam An
Assignee: Sam An


I observe in a cluster that HS2 connection to HMS was timed out. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23424) Remove Dependency on Log4J

[David Mollitor (Jira)]

David Mollitor created HIVE-23424:
-

 Summary: Remove Dependency on Log4J
 Key: HIVE-23424
 URL: https://issues.apache.org/jira/browse/HIVE-23424
 Project: Hive
  Issue Type: Improvement
Reporter: David Mollitor
Assignee: David Mollitor
 Attachments: HIVE-23424.1.patch

The project uses SLF4J but not the log4j specific libraries.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23423) Hash aggregation is always disabled in vectorized execution of grouping set queries

[Nita Dembla (Jira)]

Nita Dembla created HIVE-23423:
--

 Summary: Hash aggregation is always disabled in vectorized 
execution of grouping set queries
 Key: HIVE-23423
 URL: https://issues.apache.org/jira/browse/HIVE-23423
 Project: Hive
  Issue Type: Bug
  Components: Hive, Operators, Query Processor
Affects Versions: 4.0.0
Reporter: Nita Dembla


https://issues.apache.org/jira/browse/HIVE-23356 fixed the issue with disabling 
hash aggregation on grouping set queries. Need a fix for VectorGroupbyOperator 
operator.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23422) Investigate why skewjoin_mapjoin11.q result set has changed

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23422:
-

 Summary: Investigate why skewjoin_mapjoin11.q result set has 
changed
 Key: HIVE-23422
 URL: https://issues.apache.org/jira/browse/HIVE-23422
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely


Check [https://reviews.apache.org/r/72485/] for details.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23421) Use Liquibase for Database Schema Versioning and Upgrades

[David Mollitor (Jira)]

David Mollitor created HIVE-23421:
-

 Summary: Use Liquibase for Database Schema Versioning and Upgrades
 Key: HIVE-23421
 URL: https://issues.apache.org/jira/browse/HIVE-23421
 Project: Hive
  Issue Type: New Feature
Reporter: David Mollitor


https://github.com/liquibase/liquibase

bq. Liquibase helps millions of teams track, version, and deploy database 
schema changes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23420) Investigate why data has changed in parquet_timestamp_staging_2.q

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23420:
-

 Summary: Investigate why data has changed in 
parquet_timestamp_staging_2.q
 Key: HIVE-23420
 URL: https://issues.apache.org/jira/browse/HIVE-23420
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely


Check [https://reviews.apache.org/r/72485] for detals.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23419) Find out how to test CLUSTER BY, DISTRIBUTE BY, SORT BY in q files like parenthesis_star_by.q without having an indeterministic output

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23419:
-

 Summary: Find out how to test CLUSTER BY, DISTRIBUTE BY, SORT BY 
in q files like parenthesis_star_by.q without having an indeterministic output
 Key: HIVE-23419
 URL: https://issues.apache.org/jira/browse/HIVE-23419
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely


Check [https://reviews.apache.org/r/72485/] for details.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23418) Investigate why msck command found different partitions at repair.q

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23418:
-

 Summary: Investigate why msck command found different partitions 
at repair.q
 Key: HIVE-23418
 URL: https://issues.apache.org/jira/browse/HIVE-23418
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23417) Investigate why statistics for double type columns were modified at reduceSinkDeDuplication_pRS_key_empty.q

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23417:
-

 Summary: Investigate why statistics for double type columns were 
modified at reduceSinkDeDuplication_pRS_key_empty.q
 Key: HIVE-23417
 URL: https://issues.apache.org/jira/browse/HIVE-23417
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely


Check [https://reviews.apache.org/r/72485/#comment309242] for details.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23416) Investigate why predicate modified at quotedid_skew.q

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23416:
-

 Summary: Investigate why predicate modified at quotedid_skew.q
 Key: HIVE-23416
 URL: https://issues.apache.org/jira/browse/HIVE-23416
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely


Check [https://reviews.apache.org/r/72485/#comment309242] for details.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23415) Investigate why vectorization is not used in query_result_fileformat.q

[Miklos Gergely (Jira)]

Miklos Gergely created HIVE-23415:
-

 Summary: Investigate why vectorization is not used in 
query_result_fileformat.q
 Key: HIVE-23415
 URL: https://issues.apache.org/jira/browse/HIVE-23415
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely


Check [https://reviews.apache.org/r/72485/#comment309242] for details.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23414) Detail Hive Java Compatibility

[David Mollitor (Jira)]

David Mollitor created HIVE-23414:
-

 Summary: Detail Hive Java Compatibility
 Key: HIVE-23414
 URL: https://issues.apache.org/jira/browse/HIVE-23414
 Project: Hive
  Issue Type: Improvement
  Components: Documentation
Reporter: David Mollitor
Assignee: David Mollitor






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23413) Create a new config to skip all locks

[Peter Varga (Jira)]

Peter Varga created HIVE-23413:
--

 Summary: Create a new config to skip all locks
 Key: HIVE-23413
 URL: https://issues.apache.org/jira/browse/HIVE-23413
 Project: Hive
  Issue Type: Improvement
Reporter: Peter Varga
Assignee: Peter Varga


>From time-to-time some query is blocked on locks which should not.

To have a quick workaround for this we should have a config which the user can 
set in the session to disable acquiring/checking locks, so we can provide it 
immediately and then later investigate and fix the root cause.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23412) Filter applied in the "LEFT JOIN" condition is pulling a null batch on last reducer (map join vectorizer operator) and throwing exception as "Hive Runtime Error while pro

[kumar ravi (Jira)]

kumar ravi created HIVE-23412:
-

 Summary: Filter applied in the "LEFT JOIN" condition is pulling a 
null batch on last reducer (map join vectorizer operator) and throwing 
exception as "Hive Runtime Error while processing vector batch" at the map join 
during the query execution.
 Key: HIVE-23412
 URL: https://issues.apache.org/jira/browse/HIVE-23412
 Project: Hive
  Issue Type: Bug
Reporter: kumar ravi


Filter applied in the "LEFT JOIN" condition is pulling a null batch on last 
reducer (map join vectorizer operator) and throwing exception as "Hive Runtime 
Error while processing vector batch" at the map join during the query execution.

{{ERROR : FAILED: Execution Error, return code 2 from 
org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex re-running, vertexName=Map 
2, vertexId=vertex_1588769098286_8402_1_00Vertex re-running, vertexName=Map 1, 
vertexId=vertex_1588769098286_8402_1_02Vertex re-running, vertexName=Map 4, 
vertexId=vertex_1588769098286_8402_1_04Vertex failed, vertexName=Reducer 6, 
vertexId=vertex_1588769098286_8402_1_05, diagnostics=[Task failed, 
taskId=task_1588769098286_8402_1_05_000604, diagnostics=[TaskAttempt 0 failed, 
info=[Error: Error while running task ( failure ) : 
attempt_1588769098286_8402_1_05_000604_0:java.lang.RuntimeException: 
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum 
5)}}{{ at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:296)}}{{
 at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)}}{{ 
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)}}{{
 at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)}}{{
 at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)}}{{
 at java.security.AccessController.doPrivileged(Native Method)}}{{ at 
javax.security.auth.Subject.doAs(Subject.java:422)}}{{ at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)}}{{
 at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)}}{{
 at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)}}{{
 at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)}}{{ at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)}}{{
 at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)}}{{
 at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)}}{{
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}{{
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}{{
 at java.lang.Thread.run(Thread.java:748)}}{{Caused by: 
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum 
5)}}{{ at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:401)}}{{
 at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:249)}}{{
 at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:318)}}{{
 at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)}}{{
 ... 16 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing vector batch (tag=3) (vectorizedVertexNum 
5)}}{{ at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:503)}}{{
 at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:392)}}{{
 ... 19 more}}{{Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
Unexpected exception from VectorMapJoinOperator : null}}{{ at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:539)}}{{
 at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapJoinOperator.process(VectorMapJoinOperator.java:235)}}{{
 at 
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:490)}}{{
 ... 20 more}}{{Caused by: java.lang.NullPointerException}}{{ at 
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilterTag(CommonJoinOperator.java:802)}}{{
 at 
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genObject(CommonJoinOperator.java:600)}}{{
 at 
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genJoinObject(CommonJoinOperator.java:533)}}{{
 at 
org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.jav

Re: Review Request 72485: Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 3

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72485/#review220693
---




ql/src/test/results/clientpositive/llap/metadataOnlyOptimizer.q.out
Line 45 (original)


resultset changed: there are a lot of rows deleted; but no new one is added.



ql/src/test/results/clientpositive/llap/select_same_col.q.out
Line 25 (original)


result order have changed - no orderby



ql/src/test/results/clientpositive/llap/setop_no_distinct.q.out
Line 47 (original)


resultset order change (no oredrby)



ql/src/test/results/clientpositive/llap/skewjoin_mapjoin1.q.out
Line 132 (original), 100 (patched)


the earlier filter seems like it was stronger



ql/src/test/results/clientpositive/llap/skewjoin_mapjoin1.q.out
Line 216 (original), 171 (patched)


the earlier filter predicate was better



ql/src/test/results/clientpositive/llap/skewjoin_mapjoin10.q.out
Line 551 (original), 419 (patched)


less selective filter predicate



ql/src/test/results/clientpositive/llap/skewjoin_mapjoin11.q.out
Line 179 (original)


resultset change; from 6 rows to 1



ql/src/test/results/clientpositive/llap/skewjoin_mapjoin7.q.out
Line 215 (original), 113 (patched)


filter predicate is worse than before



ql/src/test/results/clientpositive/llap/skewjoinopt1.q.out
Line 239 (original), 179 (patched)


filter predicate is worse than before



ql/src/test/results/clientpositive/llap/skewjoinopt10.q.out
Line 66 (original), 70 (patched)


filter predicate is worse


- Zoltan Haindrich


On May 7, 2020, 7:04 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72485/
> ---
> 
> (Updated May 7, 2020, 7:04 p.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez, John Sherman, Zoltan 
> Haindrich, Krisztian Kasa, Steve Carlin, and Vineet Garg.
> 
> 
> Bugs: HIVE-23403
> https://issues.apache.org/jira/browse/HIVE-23403
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Move q tests to TestMiniLlapLocal from TestCliDriver where the output is 
> different, batch 3
> 
> 
> Diffs
> -
> 
>   ql/src/test/results/clientpositive/llap/mergejoins_mixed.q.out 04bb90c370 
>   ql/src/test/results/clientpositive/llap/metadataOnlyOptimizer.q.out 
> 1671c6b8b4 
>   ql/src/test/results/clientpositive/llap/mm_buckets.q.out e2c31637fa 
>   ql/src/test/results/clientpositive/llap/msck_repair_0.q.out 94da7c3aaf 
>   ql/src/test/results/clientpositive/llap/msck_repair_1.q.out 5f94246e67 
>   ql/src/test/results/clientpositive/llap/msck_repair_2.q.out 90f77b7cde 
>   ql/src/test/results/clientpositive/llap/msck_repair_3.q.out c18da6f437 
>   ql/src/test/results/clientpositive/llap/msck_repair_acid.q.out 902a4b7d80 
>   ql/src/test/results/clientpositive/llap/msck_repair_batchsize.q.out 
> bedfac7d28 
>   ql/src/test/results/clientpositive/llap/msck_repair_drop.q.out 04179f3304 
>   ql/src/test/results/clientpositive/llap/multi_insert_distinct.q.out 
> eefa1e1197 
>   ql/src/test/results/clientpositive/llap/multi_insert_gby.q.out d36dc8de00 
>   ql/src/test/results/clientpositive/llap/multi_insert_gby2.q.out c3db38642b 
>   ql/src/test/results/clientpositive/llap/multi_insert_gby3.q.out 23518f7ac2 
>   ql/src/test/results/clientpositive/llap/multi_insert_gby4.q.out abb749b78b 
>   ql/src/test/results/clientpositive/llap/multi_insert_mixed.q.out b7b721e500 
>   
> ql/src/test/results/clientpositive/llap/multi_insert_move_tasks_share_dependencies.q.out
>  4e34c11af0 
>   ql/src/test/results/clientpositive/llap/multi_insert_union_src.q.out 
> 90597f37d9 
>   ql/src/test/results/clientpositive/llap/multi_insert_with_join2.q.out 
> bdb876e618 
>   ql/src/test/results/clientpositive/llap/multi_join_union.q.out ac3fd77714 
>   ql/src/test/results/clientpositive/llap/multigroupby_singlemr.q.out 
> 3ae1152645 
>   ql/src/test/results/clientpositive/llap/named_column_join.q.out 9c0250e5e5 
>   ql/src/test/results/clientpositive/llap/nested_column_pruning.q.out 
> 233995910c 
>   ql/src/test/results/clientpositive/llap/no_hooks.q.out 7583863800 
>   ql/src/test/results/clientpositive/llap/noalias_subq1.q.out 7cbb6ac993 
>   ql/src/test/results/clientpositive/llap/nonblock_op_d

[jira] [Created] (HIVE-23411) conditional aggregation in Windowing and Analytics Functions

[BilboDai (Jira)]

BilboDai created HIVE-23411:
---

 Summary: conditional aggregation in Windowing and Analytics 
Functions 
 Key: HIVE-23411
 URL: https://issues.apache.org/jira/browse/HIVE-23411
 Project: Hive
  Issue Type: Improvement
Reporter: BilboDai


{{select }}{{sum(if(datediff(#order_date,order_date) <= 30, price, 0) 
over(partition by city_code order by order_date) as total_price_within_30days}}

sql above is trying to aggregate each city's total order price within 30 days 
according to each city's daily record. is it possible to support that 
conditional aggregation in window function, _#order_date_ means the order date 
of current row which being processed. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72483: "show tables like" support for SQL wildcard characters (% and _)

[Miklos Gergely]

> On May 8, 2020, 12:13 p.m., Zoltan Haindrich wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/status/ShowTableStatusOperation.java
> > Line 63 (original), 63 (patched)
> > 
> >
> > seems like earlier the filtering was done on the metastore side?
> > in case directsql is on it will collect all tablenames using a simple 
> > sql stmt...so I think this might be ok.

Yes, unfortunately the output of UDFLike.likePatternToRegExp is not acceptable 
for datanucleus's match like, as it is not supporting the full spectrum of the 
java Pattern syntax.


> On May 8, 2020, 12:13 p.m., Zoltan Haindrich wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/ddl/view/show/ShowViewsOperation.java
> > Line 56 (original), 63 (patched)
> > 
> >
> > note: at this point viewNames is already sorted...treeset is redundant 
> > here

Correct, I'll remove it.


- Miklos


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72483/#review220691
---


On May 7, 2020, 11:16 a.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72483/
> ---
> 
> (Updated May 7, 2020, 11:16 a.m.)
> 
> 
> Review request for hive and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-23359
> https://issues.apache.org/jira/browse/HIVE-23359
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Follow the SQL conventions.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/database/show/ShowDatabasesOperation.java
>  625a48ee71 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/status/ShowTableStatusOperation.java
>  914e63d80c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesOperation.java
>  4846d2969c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/show/ShowMaterializedViewsOperation.java
>  792b352e4c 
>   ql/src/java/org/apache/hadoop/hive/ql/ddl/view/show/ShowViewsOperation.java 
> 7962551cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f6a5207ddb 
>   ql/src/test/queries/clientnegative/show_tablestatus.q 283b5836e2 
>   ql/src/test/queries/clientpositive/alter1.q 0843351996 
>   ql/src/test/queries/clientpositive/alter2.q 6586afca8f 
>   ql/src/test/queries/clientpositive/alter3.q a38643de41 
>   ql/src/test/queries/clientpositive/alter4.q 7a87113edc 
>   ql/src/test/queries/clientpositive/alter5.q 5abd14f075 
>   ql/src/test/queries/clientpositive/create_view.q 392ebf4591 
>   ql/src/test/queries/clientpositive/describe_table_json.q 8aea7f283f 
>   ql/src/test/queries/clientpositive/encryption_auto_purge_tables.q 
> 14c7f7eb96 
>   ql/src/test/queries/clientpositive/encryption_drop_table.q 884e510a92 
>   ql/src/test/queries/clientpositive/encryption_move_tbl.q b63607641a 
>   ql/src/test/queries/clientpositive/input2.q c7b0d3024a 
>   ql/src/test/queries/clientpositive/input3.q 07d75bdb29 
>   ql/src/test/queries/clientpositive/rename_column.q 82036f68b0 
>   ql/src/test/queries/clientpositive/show_materialized_views.q 81f86a7a95 
>   ql/src/test/queries/clientpositive/show_tables.q 8576daa689 
>   ql/src/test/queries/clientpositive/show_tablestatus.q d8f04ec30d 
>   ql/src/test/queries/clientpositive/show_views.q 1af89b6d9e 
>   ql/src/test/queries/clientpositive/temp_table_names.q bac26d3bf3 
>   ql/src/test/queries/clientpositive/temp_table_truncate.q 93a54d9f46 
>   ql/src/test/results/clientnegative/show_tablestatus.q.out ed962e7eca 
>   ql/src/test/results/clientpositive/create_view.q.out 7414d4749d 
>   
> ql/src/test/results/clientpositive/encrypted/encryption_auto_purge_tables.q.out
>  1d7707cc76 
>   ql/src/test/results/clientpositive/encrypted/encryption_drop_table.q.out 
> 90ec23641f 
>   ql/src/test/results/clientpositive/encrypted/encryption_move_tbl.q.out 
> ff3bda628c 
>   ql/src/test/results/clientpositive/llap/alter1.q.out 1fef592df4 
>   ql/src/test/results/clientpositive/llap/alter2.q.out b9d1b6170a 
>   ql/src/test/results/clientpositive/llap/alter3.q.out 7ceeb3af9c 
>   ql/src/test/results/clientpositive/llap/alter4.q.out 74e2dfb6b6 
>   ql/src/test/results/clientpositive/llap/alter5.q.out 61a04f2702 
>   ql/src/test/results/clientpositive/llap/describe_table_json.q.out 
> cc33e6dccc 
>   ql/src/test/results/clientpositive/llap/input2.q.out 28f7da4f02 
>   ql/src/test/results/clientpositive/llap/input3.q.out 0365ff25ba 
>   ql/src/test/results/clientpositive/llap/rename_column.q.out 15816bbf98 
>   ql/src/test/results/clientpositive/llap/show_materialized_views.q.out 
> d377f97f14 
>   ql/src/test/results/clien

Re: Review Request 72483: "show tables like" support for SQL wildcard characters (% and _)

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72483/#review220691
---




ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/status/ShowTableStatusOperation.java
Line 63 (original), 63 (patched)


seems like earlier the filtering was done on the metastore side?
in case directsql is on it will collect all tablenames using a simple sql 
stmt...so I think this might be ok.



ql/src/java/org/apache/hadoop/hive/ql/ddl/view/show/ShowViewsOperation.java
Line 56 (original), 63 (patched)


note: at this point viewNames is already sorted...treeset is redundant here


- Zoltan Haindrich


On May 7, 2020, 1:16 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72483/
> ---
> 
> (Updated May 7, 2020, 1:16 p.m.)
> 
> 
> Review request for hive and Zoltan Haindrich.
> 
> 
> Bugs: HIVE-23359
> https://issues.apache.org/jira/browse/HIVE-23359
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Follow the SQL conventions.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/database/show/ShowDatabasesOperation.java
>  625a48ee71 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/status/ShowTableStatusOperation.java
>  914e63d80c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/table/info/show/tables/ShowTablesOperation.java
>  4846d2969c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/show/ShowMaterializedViewsOperation.java
>  792b352e4c 
>   ql/src/java/org/apache/hadoop/hive/ql/ddl/view/show/ShowViewsOperation.java 
> 7962551cc9 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java f6a5207ddb 
>   ql/src/test/queries/clientnegative/show_tablestatus.q 283b5836e2 
>   ql/src/test/queries/clientpositive/alter1.q 0843351996 
>   ql/src/test/queries/clientpositive/alter2.q 6586afca8f 
>   ql/src/test/queries/clientpositive/alter3.q a38643de41 
>   ql/src/test/queries/clientpositive/alter4.q 7a87113edc 
>   ql/src/test/queries/clientpositive/alter5.q 5abd14f075 
>   ql/src/test/queries/clientpositive/create_view.q 392ebf4591 
>   ql/src/test/queries/clientpositive/describe_table_json.q 8aea7f283f 
>   ql/src/test/queries/clientpositive/encryption_auto_purge_tables.q 
> 14c7f7eb96 
>   ql/src/test/queries/clientpositive/encryption_drop_table.q 884e510a92 
>   ql/src/test/queries/clientpositive/encryption_move_tbl.q b63607641a 
>   ql/src/test/queries/clientpositive/input2.q c7b0d3024a 
>   ql/src/test/queries/clientpositive/input3.q 07d75bdb29 
>   ql/src/test/queries/clientpositive/rename_column.q 82036f68b0 
>   ql/src/test/queries/clientpositive/show_materialized_views.q 81f86a7a95 
>   ql/src/test/queries/clientpositive/show_tables.q 8576daa689 
>   ql/src/test/queries/clientpositive/show_tablestatus.q d8f04ec30d 
>   ql/src/test/queries/clientpositive/show_views.q 1af89b6d9e 
>   ql/src/test/queries/clientpositive/temp_table_names.q bac26d3bf3 
>   ql/src/test/queries/clientpositive/temp_table_truncate.q 93a54d9f46 
>   ql/src/test/results/clientnegative/show_tablestatus.q.out ed962e7eca 
>   ql/src/test/results/clientpositive/create_view.q.out 7414d4749d 
>   
> ql/src/test/results/clientpositive/encrypted/encryption_auto_purge_tables.q.out
>  1d7707cc76 
>   ql/src/test/results/clientpositive/encrypted/encryption_drop_table.q.out 
> 90ec23641f 
>   ql/src/test/results/clientpositive/encrypted/encryption_move_tbl.q.out 
> ff3bda628c 
>   ql/src/test/results/clientpositive/llap/alter1.q.out 1fef592df4 
>   ql/src/test/results/clientpositive/llap/alter2.q.out b9d1b6170a 
>   ql/src/test/results/clientpositive/llap/alter3.q.out 7ceeb3af9c 
>   ql/src/test/results/clientpositive/llap/alter4.q.out 74e2dfb6b6 
>   ql/src/test/results/clientpositive/llap/alter5.q.out 61a04f2702 
>   ql/src/test/results/clientpositive/llap/describe_table_json.q.out 
> cc33e6dccc 
>   ql/src/test/results/clientpositive/llap/input2.q.out 28f7da4f02 
>   ql/src/test/results/clientpositive/llap/input3.q.out 0365ff25ba 
>   ql/src/test/results/clientpositive/llap/rename_column.q.out 15816bbf98 
>   ql/src/test/results/clientpositive/llap/show_materialized_views.q.out 
> d377f97f14 
>   ql/src/test/results/clientpositive/llap/show_tables.q.out ade1690e89 
>   ql/src/test/results/clientpositive/llap/show_tablestatus.q.out f875778430 
>   ql/src/test/results/clientpositive/llap/show_views.q.out b5d2027420 
>   ql/src/test/results/clientpositive/llap/temp_table_names.q.out f8ad01a2d9 
>   ql/src/test/results/clientpositive/llap/temp_table_truncate.q.out 
> 20aeafc2db 
> 
> 
> Diff: https://reviews.apache.org/r/7

Re: Review Request 72481: HIVE-23234: Optimize TxnHandler::allocateTableWriteIds

[Marton Bod]

> On May 8, 2020, 10:23 a.m., Peter Vary wrote:
> > Thanks Marci,
> > Few querstions below - probably I just do not understand this part of the 
> > code enough.
> > 
> > Another question for the perf test: How many threads are you using?
> > 
> > Thanks,
> > Peter

Thanks Peti for the review. See my answers below, they are related to minor 
housekeeping changes that are not core to the optimization story.

The perf tests were run with 8 concurrent threads.


> On May 8, 2020, 10:23 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
> > Line 1102 (original)
> > 
> >
> > Why did we remove this?

checkRetryable never throws MetaException (and hence the code never enters this 
catch block), so I removed it from its throws clause


> On May 8, 2020, 10:23 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
> > Line 1137 (original)
> > 
> >
> > why did we remove this?

same as above


> On May 8, 2020, 10:23 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Lines 1762-1765 (original), 1759-1762 (patched)
> > 
> >
> > Is this a functionality or performance change?

neither really, just some readability refactor


> On May 8, 2020, 10:23 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Line 2021 (original), 2013 (patched)
> > 
> >
> > Why is this change required?

just seems counterintuitive to use string concat if we're already using a 
stringbuilder anyway


> On May 8, 2020, 10:23 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Line 2067 (original), 2057 (patched)
> > 
> >
> > Why is this change?

this was causing a checkstyle issue (line lenght too long)


> On May 8, 2020, 10:23 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Line 2079 (original), 2070 (patched)
> > 
> >
> > Why is this change?

unnecessary call to Long.toString


> On May 8, 2020, 10:23 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Line 2090 (original), 2081 (patched)
> > 
> >
> > Why is this change?

same as above


- Marton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72481/#review220689
---


On May 7, 2020, 3:55 p.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72481/
> ---
> 
> (Updated May 7, 2020, 3:55 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Removed global mutex on writeId allocation, which means write ids can now be 
> allocated concurrently for different tables without blocking each other, 
> speeding up execution (perf test results below). Concurrent 
> allocateTableWriteIds() operations targeting the same table are still mutexed 
> by an S4U if the table is already present in next_write_id, otherwise a race 
> condition to insert the table into next_write_id is solved by retrying after 
> catching the duplicate key exception (the thread which commits later will be 
> the one to retry).
> 
> The situation is similar when allocateTableWriteIds() and 
> replTableWriteIdState() are running concurrently - if they target different 
> tables, they won't block each other anymore. If they target the same table, 
> and the table is already inserted into next_write_id, replTableWriteIdState() 
> returns early and allocateTableWriteIds() updates the next id. If the table 
> is not yet in next_write_id, they might attempt to insert the same row 
> concurrently, in which case who commits later will get a duplicate key 
> exception and retry the operation, just as above.
> 
> 
> Diffs
> -
> 
>   ql/src/test/org/apache/hadoop/hive/me

Re: Review Request 72481: HIVE-23234: Optimize TxnHandler::allocateTableWriteIds

[Peter Vary via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72481/#review220689
---



Thanks Marci,
Few querstions below - probably I just do not understand this part of the code 
enough.

Another question for the perf test: How many threads are you using?

Thanks,
Peter


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
Line 1102 (original)


Why did we remove this?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
Line 1137 (original)


why did we remove this?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Lines 1762-1765 (original), 1759-1762 (patched)


Is this a functionality or performance change?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 2021 (original), 2013 (patched)


Why is this change required?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 2067 (original), 2057 (patched)


Why is this change?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 2079 (original), 2070 (patched)


Why is this change?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 2090 (original), 2081 (patched)


Why is this change?


- Peter Vary


On máj. 7, 2020, 3:55 du, Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72481/
> ---
> 
> (Updated máj. 7, 2020, 3:55 du)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Removed global mutex on writeId allocation, which means write ids can now be 
> allocated concurrently for different tables without blocking each other, 
> speeding up execution (perf test results below). Concurrent 
> allocateTableWriteIds() operations targeting the same table are still mutexed 
> by an S4U if the table is already present in next_write_id, otherwise a race 
> condition to insert the table into next_write_id is solved by retrying after 
> catching the duplicate key exception (the thread which commits later will be 
> the one to retry).
> 
> The situation is similar when allocateTableWriteIds() and 
> replTableWriteIdState() are running concurrently - if they target different 
> tables, they won't block each other anymore. If they target the same table, 
> and the table is already inserted into next_write_id, replTableWriteIdState() 
> returns early and allocateTableWriteIds() updates the next id. If the table 
> is not yet in next_write_id, they might attempt to insert the same row 
> concurrently, in which case who commits later will get a duplicate key 
> exception and retry the operation, just as above.
> 
> 
> Diffs
> -
> 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 868da0c7a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  d59f863b11 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  cf41ef8aaf 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  1e177f4a7b 
> 
> 
> Diff: https://reviews.apache.org/r/72481/diff/1/
> 
> 
> Testing
> ---
> 
> Unit test in TestTxnHandler
> + Perf tests:
> dbTypesameTable variant  ms/op  error
> MYSQL FALSE original 46.93  3.041
> MYSQL FALSE patched  19.283 1.311
> MYSQL TRUE  original 50.185 3.595
> MYSQL TRUE  patched  32.254 2.164
> ORACLEFALSE original 57.609 4.461
> ORACLEFALSE patched  25.721 2.551
> ORACLETRUE  original 59.668 3.172
> ORACLETRUE  patched  39.061 2.548
> POSTGRES  FALSE original 39.364 2.94 
> POSTGRES  FALSE patched  18.518 1.038
> POSTGRES  TRUE  original 39.868 2.679
> POSTGRES  TRUE  patched  28.874 1.768
> SQLSERVER FALSE original 45.252 1.643
> SQLSERVER FALSE patched  24.583 1.529
> SQLSERVER TRUE  original 49.149 3.45 
> SQLSERVER TRUE  patched  32.918 1.654
> (sameT

Re: Review Request 72480: HIVE-23242 Fix flaky tests testHouseKeepingThreadExistence

[Peter Varga via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72480/
---

(Updated May 8, 2020, 9:46 a.m.)


Review request for hive, Miklos Gergely and Peter Vary.


Repository: hive-git


Description
---

Fix the timing to avoid flakyness.


Diffs (updated)
-

  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreHousekeepingLeaderTestBase.java
 a39a9c8e04 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/MetastoreTaskThreadAlwaysTestImpl.java
 4cd2c58896 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/RemoteMetastoreTaskThreadTestImpl1.java
 c590b6aad5 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/RemoteMetastoreTaskThreadTestImpl2.java
 5b50f66c51 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingLeader.java
 03a8161ea4 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingLeaderEmptyConfig.java
 75ea637503 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestMetastoreHousekeepingNonLeader.java
 0341d3c03b 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 7bba8d6ee6 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/MetaStoreTestUtils.java
 2702e69f86 


Diff: https://reviews.apache.org/r/72480/diff/2/

Changes: https://reviews.apache.org/r/72480/diff/1-2/


Testing
---


Thanks,

Peter Varga

[jira] [Created] (HIVE-23410) ACID: Improve the delete and update operations to avoid the move step

[Marta Kuczora (Jira)]

Marta Kuczora created HIVE-23410:


 Summary: ACID: Improve the delete and update operations to avoid 
the move step
 Key: HIVE-23410
 URL: https://issues.apache.org/jira/browse/HIVE-23410
 Project: Hive
  Issue Type: Improvement
Affects Versions: 4.0.0
Reporter: Marta Kuczora
Assignee: Marta Kuczora


This is a follow-up task for 
[HIVE-21164|https://issues.apache.org/jira/browse/HIVE-21164], where the insert 
operation has been modified to write directly to the table locations instead of 
the staging directory. The same improvement should be done for the ACID update 
and delete operations as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)