[jira] [Created] (HIVE-27137) Remove HIVE_IN_TEST_ICEBERG flag
Zsolt Miskolczi created HIVE-27137:
--
Summary: Remove HIVE_IN_TEST_ICEBERG flag
Key: HIVE-27137
URL: https://issues.apache.org/jira/browse/HIVE-27137
Project: Hive
Issue Type: Improvement
Components: Iceberg integration
Reporter: Zsolt Miskolczi
Remove the HIVE_IN_TEST_ICEBERG flag from the production code.
Remove code snippet from TxnHandler and update unit tests which are expecting
the exception.
{{
if (lc.isSetOperationType() && lc.getOperationType() == DataOperationType.UNSET
&&
((MetastoreConf.getBoolVar(conf, ConfVars.HIVE_IN_TEST) ||
MetastoreConf.getBoolVar(conf, ConfVars.HIVE_IN_TEZ_TEST)) &&
!MetastoreConf.getBoolVar(conf, ConfVars.HIVE_IN_TEST_ICEBERG))) {
}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26993) VectorizedRowBatch selected should be only modified with methods to avoid invalid object state
Zsolt Miskolczi created HIVE-26993: -- Summary: VectorizedRowBatch selected should be only modified with methods to avoid invalid object state Key: HIVE-26993 URL: https://issues.apache.org/jira/browse/HIVE-26993 Project: Hive Issue Type: Bug Components: HiveServer2, Vectorization Reporter: Zsolt Miskolczi As it is described in HIVE-26992, using field access can result invalid object state in VectorizedRowBatch. There is a quick fix in HIVE-26992 but on a long-term it will be a bullet-proof solution to not to allow field access directly. When it is solved, the extra tests added in HIVE-26992 can be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26992) Select count(*) query fails with error "java.lang.ArrayIndexOutOfBoundsException: arraycopy: last source index 1024 out of bounds for int[256]"
Zsolt Miskolczi created HIVE-26992:
--
Summary: Select count(*) query fails with error
"java.lang.ArrayIndexOutOfBoundsException: arraycopy: last source index 1024
out of bounds for int[256]"
Key: HIVE-26992
URL: https://issues.apache.org/jira/browse/HIVE-26992
Project: Hive
Issue Type: Bug
Components: llap
Reporter: Zsolt Miskolczi
{{java.lang.RuntimeException: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:351)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:101)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:437)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:297)
... 15 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:970)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
... 18 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: last source
index 1024 out of bounds for int[256]
at java.base/java.lang.System.arraycopy(Native Method)
at
org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:116)
at
org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:968)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:172)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.deliverVectorizedRowBatch(VectorMapOperator.java:809)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:842)
... 19 more
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
attempt_1674594639075_0001_39_00_02_1:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:351)
at
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:280)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:82)
at
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:69)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:69)
at
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:39)
at org.apache.tez.common.CallableWithNdc.call(Callabl
[jira] [Created] (HIVE-26990) Upgrade Iceberg to 1.1.0
Zsolt Miskolczi created HIVE-26990: -- Summary: Upgrade Iceberg to 1.1.0 Key: HIVE-26990 URL: https://issues.apache.org/jira/browse/HIVE-26990 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Zsolt Miskolczi Assignee: Zsolt Miskolczi Iceberg integration currently uses Iceberg 1.0.0 Upgrade it to 1.1.0 to be able to utilise new features. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26834) Hive Iceberg Storage Handler tests are ignored
Zsolt Miskolczi created HIVE-26834: -- Summary: Hive Iceberg Storage Handler tests are ignored Key: HIVE-26834 URL: https://issues.apache.org/jira/browse/HIVE-26834 Project: Hive Issue Type: Test Components: HiveServer2, Iceberg integration Reporter: Zsolt Miskolczi I wanted to run the following test locally: `mvn test -Dtest="TestHiveIcebergStorageHandlerNoScan#testIcebergAndHmsTableProperties"` And it was just skipped. I have checked the latest run on the CI server and it ignored it as well: Link: http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/1527/artifacts/ ``` [2022-12-12T09:11:50.841Z] [INFO] Running org.apache.iceberg.mr.hive.TestHiveIcebergStorageHandlerNoScan [2022-12-12T09:11:50.886Z] [INFO] No tests to run. ``` -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26823) Port spotlessApply changes
Zsolt Miskolczi created HIVE-26823: -- Summary: Port spotlessApply changes Key: HIVE-26823 URL: https://issues.apache.org/jira/browse/HIVE-26823 Project: Hive Issue Type: Sub-task Reporter: Zsolt Miskolczi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26824) Port changes after spotlessApply
Zsolt Miskolczi created HIVE-26824: -- Summary: Port changes after spotlessApply Key: HIVE-26824 URL: https://issues.apache.org/jira/browse/HIVE-26824 Project: Hive Issue Type: Sub-task Reporter: Zsolt Miskolczi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26822) Port changes before spotlessApply
Zsolt Miskolczi created HIVE-26822: -- Summary: Port changes before spotlessApply Key: HIVE-26822 URL: https://issues.apache.org/jira/browse/HIVE-26822 Project: Hive Issue Type: Sub-task Reporter: Zsolt Miskolczi -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26808) Port Iceberg catalog changes
Zsolt Miskolczi created HIVE-26808: -- Summary: Port Iceberg catalog changes Key: HIVE-26808 URL: https://issues.apache.org/jira/browse/HIVE-26808 Project: Hive Issue Type: Improvement Components: Iceberg integration Reporter: Zsolt Miskolczi The last round of porting happened in 2022 april, there were a couple of changes especially in HiveTableOperations worth porting into iceberg-catalog. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26798) Revert HIVE-26763
Zsolt Miskolczi created HIVE-26798: -- Summary: Revert HIVE-26763 Key: HIVE-26798 URL: https://issues.apache.org/jira/browse/HIVE-26798 Project: Hive Issue Type: Bug Components: Testing Infrastructure Reporter: Zsolt Miskolczi Reason: https://issues.apache.org/jira/browse/HIVE-26796 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26797) Tests are excluded after Zookeeper and curator upgrade
Zsolt Miskolczi created HIVE-26797: -- Summary: Tests are excluded after Zookeeper and curator upgrade Key: HIVE-26797 URL: https://issues.apache.org/jira/browse/HIVE-26797 Project: Hive Issue Type: Bug Components: HiveServer2, Standalone Metastore, Tests Reporter: Zsolt Miskolczi Attachments: screenshot-1.png https://issues.apache.org/jira/browse/HIVE-26763 upgraded Zookeeper and curator. After that, we have noticed `StartMiniHS2Cluster` doesn't run at all. It doesn't throw an error. It just excluded. The whole hive-unit is affected as well so that I made a comparison between the Zookeeper change test run and the previous precommit test run before that: - The Zookeeper update ran 42430 tests: http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3787/7/tests/ - The previous change ran 46838: http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3784/3/tests/ It is about 4000 test cases as a difference. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26482) Create a unit test checking compaction output file names on a partitioned table
Zsolt Miskolczi created HIVE-26482:
--
Summary: Create a unit test checking compaction output file names
on a partitioned table
Key: HIVE-26482
URL: https://issues.apache.org/jira/browse/HIVE-26482
Project: Hive
Issue Type: Test
Components: Hive
Reporter: Zsolt Miskolczi
Assignee: Zsolt Miskolczi
Compaction output directories' writeIds only reflect the writeIds of the deltas
it compacts, and not the max write id of the table
Example:
Pre-compaction...
{code:java}
Partition p=1 contains:
delta_1_1
delta_2_2
partition p=2 contains
delta_3_3
delta_4_4
{code}
After minor compaction...
{code:java}
Partition p=1 contains:
delta_1_2
partition p=2 contains
delta_3_4
{code}
AFAIK there are no unit tests that reflect this.
TestTxnCommands2#testFullACIDAbortWithManyPartitions is a good template to
start with.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26455) Remove PowerMockito from hive-exec
Zsolt Miskolczi created HIVE-26455: -- Summary: Remove PowerMockito from hive-exec Key: HIVE-26455 URL: https://issues.apache.org/jira/browse/HIVE-26455 Project: Hive Issue Type: Improvement Components: Hive Reporter: Zsolt Miskolczi PowerMockito is a mockito extension that introduces some painful points. The main intention behind that is to be able to do static mocking. Since its release, mockito-inline has been released, as a part of the mockito-core. It doesn't require vintage test runner to be able to run and it can mock objects with their own thread. The goal is to stop using PowerMockito and use mockito-inline instead. The affected packages are: * org.apache.hadoop.hive.ql.exec.repl * org.apache.hadoop.hive.ql.exec.repl.bootstrap.load * org.apache.hadoop.hive.ql.exec.repl.ranger; * org.apache.hadoop.hive.ql.exec.util * org.apache.hadoop.hive.ql.parse.repl * org.apache.hadoop.hive.ql.parse.repl.load.message * org.apache.hadoop.hive.ql.parse.repl.metric * org.apache.hadoop.hive.ql.txn.compactor -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-26407) Do not collect statistics if the compaction fails
Zsolt Miskolczi created HIVE-26407:
--
Summary: Do not collect statistics if the compaction fails
Key: HIVE-26407
URL: https://issues.apache.org/jira/browse/HIVE-26407
Project: Hive
Issue Type: Test
Components: Hive
Reporter: Zsolt Miskolczi
It can still compute statistics, even if compaction fails.
if (computeStats) \{
StatsUpdater.gatherStats(ci, conf, runJobAsSelf(ci.runAs) ? ci.runAs :
t1.getOwner(),
CompactorUtil.getCompactorJobQueueName(conf, ci, t1));
}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-26392) Move StringTableMap tests into a dedicated test class
Zsolt Miskolczi created HIVE-26392: -- Summary: Move StringTableMap tests into a dedicated test class Key: HIVE-26392 URL: https://issues.apache.org/jira/browse/HIVE-26392 Project: Hive Issue Type: Test Components: Hive Reporter: Zsolt Miskolczi `StringTableMap` has unit tests in `TestWorker.java`. They could be in their own dedicated test class instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
