[jira] [Created] (HIVE-25390) Metrics compaction_failed_initiator_ratio and compaction_failed_cleaner_ratio should be counters

[Karen Coppage (Jira)]

Karen Coppage created HIVE-25390:


 Summary: Metrics compaction_failed_initiator_ratio and 
compaction_failed_cleaner_ratio should be counters
 Key: HIVE-25390
 URL: https://issues.apache.org/jira/browse/HIVE-25390
 Project: Hive
  Issue Type: Sub-task
Reporter: Karen Coppage
Assignee: Karen Coppage


Metric compaction_failed_initiator_ratio represents the ratio of initiator 
failures to the total number of initiator runs, both computed since Metastore 
was restarted, represented by a double. This isn't really useable. It would be 
better if it counted the number of initiator failures since Metastore was 
restarted so other components can keep an eye on things like "number of 
initiator failures in the last day".

Same goes for compaction_failed_cleaner_ratio.

This commit removes metrics
 * compaction_failed_initiator_ratio
 * compaction_failed_cleaner_ratio

and introduces metrics
 * compaction_initiator_failure_counter
 * compaction_cleaner_failure_counter



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25391) Turn off priv synchronizer by default

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-25391:
---

 Summary: Turn off priv synchronizer by default
 Key: HIVE-25391
 URL: https://issues.apache.org/jira/browse/HIVE-25391
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


In large deployments, good set of scale issues were identified with priv 
synchronizer. It will be good to turn it off by default, and let the users 
enable it on need basis.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25392) Refactor UDFToInteger to GenericUDFToInteger

[Ashish Sharma (Jira)]

Ashish Sharma created HIVE-25392:


 Summary: Refactor UDFToInteger to GenericUDFToInteger
 Key: HIVE-25392
 URL: https://issues.apache.org/jira/browse/HIVE-25392
 Project: Hive
  Issue Type: Sub-task
Reporter: Ashish Sharma
Assignee: Ashish Sharma


Refactor UDFToInteger to move from UDF to GenericUDF.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25393) Prevent concurrent compaction and DDL operations

[Denys Kuzmenko (Jira)]

Denys Kuzmenko created HIVE-25393:
-

 Summary: Prevent concurrent compaction and DDL operations
 Key: HIVE-25393
 URL: https://issues.apache.org/jira/browse/HIVE-25393
 Project: Hive
  Issue Type: Task
Reporter: Denys Kuzmenko


Introduce read locks in compaction (MR, Minor Query-based) that would prevent 
concurrency issues with DDL operations (DROP/RENAME table).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25394) Enable vectorization for TestIcebergCliDriver dynamic_partition_pruning.q

[Peter Vary (Jira)]

Peter Vary created HIVE-25394:
-

 Summary: Enable vectorization for TestIcebergCliDriver 
dynamic_partition_pruning.q
 Key: HIVE-25394
 URL: https://issues.apache.org/jira/browse/HIVE-25394
 Project: Hive
  Issue Type: Test
Reporter: Peter Vary


If we turn on vectorization for {{dynamic_partition_pruning.q}} we will get the 
following exception:
{code}
See ./ql/target/tmp/log/hive.log or ./itests/qtest/target/tmp/log/hive.log, or 
check ./ql/target/surefire-reports or ./itests/qtest/target/surefire-reports/ 
for specific test cases logs.
 org.apache.hadoop.hive.ql.metadata.HiveException: Vertex failed, 
vertexName=Map 1, vertexId=vertex_1627387142352_0001_11_01, diagnostics=[Task 
failed, taskId=task_1627387142352_0001_11_01_00, diagnostics=[TaskAttempt 0 
failed, info=[Error: Error while running task ( failure ) : 
attempt_1627387142352_0001_11_01_00_0:java.lang.RuntimeException: 
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: 
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:365)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:277)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at 
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at 
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
java.io.IOException: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:89)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:414)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:311)
... 16 more
Caused by: java.io.IOException: java.lang.NullPointerException
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at 
org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:374)
at 
org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:82)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:119)
at 
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:59)
at 
org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.next(TezGroupedSplitsInputFormat.java:145)
at 
org.apache.tez.mapreduce.lib.MRReaderMapred.next(MRReaderMapred.java:116)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:75)
... 18 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatchCtx.addPartitionColsToBatch(VectorizedRowBatchCtx.java:595)
at 
org.apache.iceberg.mr.hive.vector.VectorizedRowBatchIterator.advance(VectorizedRowBatchIterator.java:69)
at 
org.apache.iceberg.mr.hive.vector.VectorizedRowBatchIterator.hasNext(VectorizedRowBatchIterator.java:81)
at 
org.apache.iceberg.mr.mapreduce.IcebergInputFormat$IcebergRecordReader.nextKeyValue(IcebergInputFormat.java:222)
at 
org.apache.iceberg.mr.hive.vector.HiveIcebergVectorizedRecordReader.next(HiveIcebergVectorizedRecordReader.java:48)
at 
org.apache.iceberg.mr.hive.vector.HiveIcebergVect

[jira] [Created] (HIVE-25395) Update hadoop to a more recent version

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-25395:
---

 Summary: Update hadoop to a more recent version
 Key: HIVE-25395
 URL: https://issues.apache.org/jira/browse/HIVE-25395
 Project: Hive
  Issue Type: Improvement
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


we are still depending on hadoop 3.1.0

which doesn't have source attachments - and makes development harder



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25396) Improve uncaught Thread Exception handling in Hive Server 2

[Matt McCline (Jira)]

Matt McCline created HIVE-25396:
---

 Summary: Improve uncaught Thread Exception handling in Hive Server 
2
 Key: HIVE-25396
 URL: https://issues.apache.org/jira/browse/HIVE-25396
 Project: Hive
  Issue Type: Bug
Reporter: Matt McCline
Assignee: Matt McCline


Hive's org.apache.hive.service.thrift.ThriftHttpServlet.doPost method does not 
handle all Exception kinds. This leaves uncaught Exception handling choices to 
the Jetty HTTP library. We fix that.

Also, a Thread.UncaughtExceptionHandler is added to Hive Server 2 so uncaught 
Exception are handled uniformly, including making them logged and not just 
printed to stderr.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)