[jira] [Created] (HIVE-25032) Optimise PartitionManagementTask
Rajesh Balamohan created HIVE-25032: --- Summary: Optimise PartitionManagementTask Key: HIVE-25032 URL: https://issues.apache.org/jira/browse/HIVE-25032 Project: Hive Issue Type: Improvement Reporter: Rajesh Balamohan When large number of tables and dbs are present, it takes lot of time to iterate over all tables in PartitionManagementTask. If table parameters are populated in TableMeta, it will cut down lots of interactions with HMS in PartitionManagementTask. Code Ref: [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L126] [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PartitionManagementTask.java#L132] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25031) Flaky test : Testing / split-01 / PostProcess / testCliDriver[orc_ppd_basic] – org.apache.hadoop.hive.cli.split1.TestMiniLlapCliDriver
Kishen Das created HIVE-25031: - Summary: Flaky test : Testing / split-01 / PostProcess / testCliDriver[orc_ppd_basic] – org.apache.hadoop.hive.cli.split1.TestMiniLlapCliDriver Key: HIVE-25031 URL: https://issues.apache.org/jira/browse/HIVE-25031 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Kishen Das java.lang.AssertionError: Client Execution succeeded but contained differences (error code = 1) after executing orc_ppd_basic.q 1795c1795 < HDFS_BYTES_READ: 0 --- > HDFS_BYTES_READ: 1759 1797c1797 < HDFS_READ_OPS: 2 --- > HDFS_READ_OPS: 3 at org.junit.Assert.fail(Assert.java:89) at org.apache.hadoop.hive.ql.QTestUtil.failedDiff(QTestUtil.java:1045) at org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:183) at org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:157) at org.apache.hadoop.hive.cli.split1.TestMiniLlapCliDriver.testCliDriver(TestMiniLlapCliDriver.java:62) at sun.reflect.GeneratedMethodAccessor307.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:135) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:27) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:95) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138) at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451) 021-04-12T23:31:19,937 ERROR [main] QTestUtil: Client Execution succeeded but contained differences (error code = 1) after executing orc_ppd_basic.q 1795c1795 < HDFS_BYTES_READ: 0 --- > HDFS_BYTES_READ: 1759 1797c1797 < HDFS_READ_OPS: 2 --- > HDFS_READ_OPS: 3 2021-04-12T23:31:19,938 INFO [main] control.CoreCliDriver: Done query orc_ppd_basic.q. succeeded=false, skipped=false. ElapsedTime(ms)=45693 Done query orc_ppd_basic.q. succeeded=false, skipped=false. ElapsedTime(ms)=45693 2021-04-12T23:31:20,044 INFO [main] zookeeper.ZooKeeper: Session: 0x100054ba2210061 closed 2021-04-12T23:31:20,049 INFO [main] zookeeper.ZooKeeper: Initiatin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25030) Flaky test : Testing / split-16 / PostProcess / testWmPoolMetricsAfterKillTrigger – org.apache.hive.jdbc.TestWMMetricsWithTrigger
Kishen Das created HIVE-25030: - Summary: Flaky test : Testing / split-16 / PostProcess / testWmPoolMetricsAfterKillTrigger – org.apache.hive.jdbc.TestWMMetricsWithTrigger Key: HIVE-25030 URL: https://issues.apache.org/jira/browse/HIVE-25030 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Kishen Das h4. Stacktrace java.lang.AssertionError: Exception while executing statement expected null, but was: at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotNull(Assert.java:756) at org.junit.Assert.assertNull(Assert.java:738) at org.apache.hive.jdbc.TestWMMetricsWithTrigger.testWmPoolMetricsAfterKillTrigger(TestWMMetricsWithTrigger.java:207) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) ound-Pool: Thread-1120] reexec.ReExecuteLostAMQueryPlugin: Got exception message: Vertex failed, vertexName=Reducer 2, vertexId=vertex_1618310614458_0001_1_02, diagnostics=[Task failed, taskId=task_1618310614458_0001_1_02_00, diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( failure ) : attempt_1618310614458_0001_1_02_00_0:java.lang.RuntimeException: java.lang.SecurityException: UDF org.apache.hive.jdbc.TestKillQueryWithAuthorizationDisabled.SleepMsUDF is not allowed at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:365) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:277) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:381) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:75) at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:62) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:62) at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:38) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:118) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.SecurityException: UDF org.apache.hive.jdbc.TestKillQueryWithAuthorizationDisabled.SleepMsUDF is not allowed at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClassInternal(GenericUDFBridge.java:146) at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.getUdfClass(GenericUDFBridge.java:133) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.isStateful(FunctionRegistry.java:1271) at org.apache.hadoop.hive.ql.exec.FunctionRegistry.isDeterministic(FunctionRegistry.java:1234) at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.isDeterministic(ExprNodeGenericFuncEvaluator.java:192) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.isConsistentWithinQuery(ExprNodeEvaluator.java:117) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.iterate(ExprNodeEvaluatorFactory.java:102) at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorFactory.toCachedEvals(ExprNodeEvaluatorFactory.java:76) at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:70) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:549) at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:503) at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:369) at org.apache.hadoop.hive.
[jira] [Created] (HIVE-25029) Remove travis builds
Zoltan Haindrich created HIVE-25029: --- Summary: Remove travis builds Key: HIVE-25029 URL: https://issues.apache.org/jira/browse/HIVE-25029 Project: Hive Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich travis only compiles the project - we already do much more than that during precommit testing. (and it it sometimes delays build because travis cant allocate executors/etc) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25028) Hive: Select query with IS operator producing unexpected result
Soumyakanti Das created HIVE-25028:
--
Summary: Hive: Select query with IS operator producing unexpected
result
Key: HIVE-25028
URL: https://issues.apache.org/jira/browse/HIVE-25028
Project: Hive
Issue Type: Bug
Components: Parser
Reporter: Soumyakanti Das
Assignee: Soumyakanti Das
Hive: Select query with IS operator is producing unexpected result.
The following was executed on postgres:
{code:java}
sqlancer=# create table if not exists emp(name text, age int);
CREATE TABLE
sqlancer=# insert into emp values ('a', 5), ('b', 15), ('c', 12);
INSERT 0 3
sqlancer=# select emp.age from emp where emp.age > 10;
age
-
15
12
(2 rows)sqlancer=# select emp.age > 10 is true from emp;
?column?
--
f
t
t
(3 rows){code}
This is happening because IS operator has higher precedence than comparison
operators in Hive. In most other databases, comparison operator has higher
precedence. The grammar needs to be changed to fix the precedence.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25027) Hide Iceberg module behind a profile
Peter Vary created HIVE-25027:
-
Summary: Hide Iceberg module behind a profile
Key: HIVE-25027
URL: https://issues.apache.org/jira/browse/HIVE-25027
Project: Hive
Issue Type: Task
Reporter: Peter Vary
Assignee: Peter Vary
After creating {{patched-iceberg-core}} and {{patched-iceberg-api}} modules the
maven build works fine, but IntelliJ needs manual classpath setup for the build
in the IntelliJ to succeed.
Most of the community does not use Iceberg and eventually the "patched" modules
will be removed as the Hive-Iceberg integration stabilizes and the Iceberg
project releases the changes we need. In the meantime we just hide the whole
{{Iceberg}} module behind a profile which is only used on the CI and if the
developer specifically sets it.
It could be used like"
{code:java}
mvn clean install -DskipTests -Piceberg{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-25026) hive sql result is duplicate data cause of same task resubmission
hezhang created HIVE-25026: -- Summary: hive sql result is duplicate data cause of same task resubmission Key: HIVE-25026 URL: https://issues.apache.org/jira/browse/HIVE-25026 Project: Hive Issue Type: Bug Components: Hive Affects Versions: 3.1.1 Reporter: hezhang Assignee: hezhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-25025) Distcp In MoveTask may cause stats info lost
WangHualei created HIVE-25025: - Summary: Distcp In MoveTask may cause stats info lost Key: HIVE-25025 URL: https://issues.apache.org/jira/browse/HIVE-25025 Project: Hive Issue Type: Bug Components: Hive Environment: example: set hive.exec.copyfile.maxsize=0; set hive.exec.copyfile.maxnumfiles=0; insert overwrite table abd_new select * from abc; select count(*) from abd_new ; select * from abd_new ; then the count(*) result will be 0, but select * will display real data, because stats info lost. Reporter: WangHualei Assignee: WangHualei after set _Run_ _as_ _end_ _user_ _instead_ _of_ _Hive_ _user_ , when execute insert overwrite , In MoveTask ,if source byte > HIVE_EXEC_COPYFILE_MAXSIZE and source file count> HIVE_EXEC_COPYFILE_MAXNUMFILES , HIve will use distcp method, it may cause tmp stats file lost. -- This message was sent by Atlassian Jira (v8.3.4#803005)
