[jira] [Work logged] (HIVE-26827) Add configs in error message to workaround predicate issue with Parquet on TIMESTAMP data type
[
https://issues.apache.org/jira/browse/HIVE-26827?focusedWorklogId=859551&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859551
]
ASF GitHub Bot logged work on HIVE-26827:
-
Author: ASF GitHub Bot
Created on: 28/Apr/23 06:30
Start Date: 28/Apr/23 06:30
Worklog Time Spent: 10m
Work Description: tarak271 commented on code in PR #4200:
URL: https://github.com/apache/hive/pull/4200#discussion_r1179981753
##
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/LeafFilterFactory.java:
##
@@ -37,6 +37,9 @@
public class LeafFilterFactory {
private static final Logger LOG =
LoggerFactory.getLogger(LeafFilterFactory.class);
+ public static final String MESSAGE =
Review Comment:
Renamed variable to FILTER_PREDICATE_CONVERSION_NOT_SUPPORTED and added
@VisibleForTesting
Issue Time Tracking
---
Worklog Id: (was: 859551)
Time Spent: 1h 20m (was: 1h 10m)
> Add configs in error message to workaround predicate issue with Parquet on
> TIMESTAMP data type
> --
>
> Key: HIVE-26827
> URL: https://issues.apache.org/jira/browse/HIVE-26827
> Project: Hive
> Issue Type: Improvement
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The below query fails with error
> {noformat}
> select * from db.parquet_table_with_timestamp where created_date_utc between
> '2022-11-05 00:01:01' and '2022-11-08 23:59:59'{noformat}
>
> We can workaround the issue below
>
> {noformat}
> 2022-11-10 06:43:36,751 [ERROR] [TezChild]
> |read.ParquetFilterPredicateConverter|: fail to build predicate filter leaf
> with errors org.apache.hadoop.hive.ql.metadata.HiveException: Conversion to
> Parquet FilterPredicate not supported for TIMESTAMP{noformat}
> by setting configs at session level
> # set hive.optimize.index.filter=false;
> # set hive.optimize.ppd=false;
> As part of this Jira proposing to add these config info to the above message
> so that who ever encounter this problem can try the workaround
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27234) Iceberg: CREATE BRANCH SQL implementation
[
https://issues.apache.org/jira/browse/HIVE-27234?focusedWorklogId=859548&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859548
]
ASF GitHub Bot logged work on HIVE-27234:
-
Author: ASF GitHub Bot
Created on: 28/Apr/23 04:46
Start Date: 28/Apr/23 04:46
Worklog Time Spent: 10m
Work Description: ayushtkn commented on code in PR #4216:
URL: https://github.com/apache/hive/pull/4216#discussion_r1179920182
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/branch/create/AlterTableCreateBranchAnalyzer.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.ddl.table.branch.create;
+
+import java.util.Locale;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.hadoop.hive.common.TableName;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ddl.DDLSemanticAnalyzerFactory;
+import org.apache.hadoop.hive.ql.ddl.DDLWork;
+import org.apache.hadoop.hive.ql.ddl.table.AbstractAlterTableAnalyzer;
+import org.apache.hadoop.hive.ql.ddl.table.AlterTableType;
+import org.apache.hadoop.hive.ql.exec.TaskFactory;
+import org.apache.hadoop.hive.ql.hooks.ReadEntity;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+import org.apache.hadoop.hive.ql.parse.AlterTableCreateBranchSpec;
+import org.apache.hadoop.hive.ql.parse.HiveParser;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+
+@DDLSemanticAnalyzerFactory.DDLType(types =
HiveParser.TOK_ALTERTABLE_CREATE_BRANCH)
+public class AlterTableCreateBranchAnalyzer extends AbstractAlterTableAnalyzer
{
+
+ public AlterTableCreateBranchAnalyzer(QueryState queryState) throws
SemanticException {
+super(queryState);
+ }
+
+ @Override
+ protected void analyzeCommand(TableName tableName, Map
partitionSpec, ASTNode command)
+ throws SemanticException {
+Table table = getTable(tableName);
+validateAlterTableType(table, AlterTableType.CREATEBRANCH, false);
+if (!"ICEBERG".equalsIgnoreCase(table.getParameters().get("table_type"))) {
+ throw new SemanticException("Cannot perform ALTER CREATE BRANCH
statement on non-iceberg table.");
+}
+inputs.add(new ReadEntity(table));
+
+String branchName = command.getChild(0).getText();
+Long snapshotId = null;
+Long maxRefAgeMs = null;
+Integer minSnapshotsToKeep = null;
+Long maxSnapshotAgeMs = null;
+for (int i = 1; i < command.getChildCount(); i++) {
+ ASTNode childNode = (ASTNode) command.getChild(i);
+ switch (childNode.getToken().getType()) {
+ case HiveParser.TOK_AS_OF_VERSION_BRANCH:
+snapshotId = Long.valueOf(childNode.getChild(0).getText());
+break;
+ case HiveParser.TOK_RETAIN:
+String maxRefAge = childNode.getChild(0).getText();
+String timeUnitOfBranchRetain = childNode.getChild(1).getText();
+maxRefAgeMs =
TimeUnit.valueOf(timeUnitOfBranchRetain.toUpperCase(Locale.ENGLISH)).toMillis(Long.valueOf(maxRefAge));
+break;
+ case HiveParser.TOK_WITH_SNAPSHOT_RETENTION:
+minSnapshotsToKeep = Integer.valueOf(childNode.getChild(0).getText());
+if (childNode.getChildren().size() > 1) {
+ String maxSnapshotAge = childNode.getChild(1).getText();
+ String timeUnitOfSnapshotsRetention =
childNode.getChild(2).getText();
+ maxSnapshotAgeMs =
TimeUnit.valueOf(timeUnitOfSnapshotsRetention.toUpperCase(Locale.ENGLISH)).toMillis(Long.valueOf(maxSnapshotAge));
Review Comment:
use. ``Long.parseLong(maxSnapshotAge)``
Issue Time Tracking
---
Worklog Id: (was: 859548)
Time Spent: 2h 40m (was: 2.5h)
> Iceberg: CREATE BRANCH SQL implementation
> --
>
> Key: HIVE-27234
> URL: https://issues.apache.org/jira/browse/HIVE-27234
> Project: Hive
> Issue Type: Sub-task
> Components: Iceberg integration
>Reporter: zhangbutao
>Assignee: zhangbutao
>
[jira] [Work logged] (HIVE-27234) Iceberg: CREATE BRANCH SQL implementation
[
https://issues.apache.org/jira/browse/HIVE-27234?focusedWorklogId=859547&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859547
]
ASF GitHub Bot logged work on HIVE-27234:
-
Author: ASF GitHub Bot
Created on: 28/Apr/23 04:46
Start Date: 28/Apr/23 04:46
Worklog Time Spent: 10m
Work Description: ayushtkn commented on code in PR #4216:
URL: https://github.com/apache/hive/pull/4216#discussion_r1179917689
##
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergBranchOperation.java:
##
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.mr.hive;
+
+import java.io.IOException;
+import java.util.concurrent.TimeUnit;
+import org.apache.iceberg.SnapshotRef;
+import org.apache.iceberg.Table;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestHiveIcebergBranchOperation extends
HiveIcebergStorageHandlerWithEngineBase {
+
+ @Test
Review Comment:
Do you need so many tests? Just create a table once and can create multiple
branches of that with different parameters ?
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##
@@ -676,6 +678,32 @@ public void
executeOperation(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
}
}
+ @Override
+ public void createBranchOperation(org.apache.hadoop.hive.ql.metadata.Table
hmsTable,
+ AlterTableCreateBranchSpec createBranchSpec) {
+TableDesc tableDesc = Utilities.getTableDesc(hmsTable);
+Table icebergTable = IcebergTableUtil.getTable(conf,
tableDesc.getProperties());
+
+String branchName = createBranchSpec.getBranchName();
+ManageSnapshots manageSnapshots = icebergTable.manageSnapshots();
+Long snapShotId = Optional.ofNullable(createBranchSpec.getSnapshotId())
+.orElse(icebergTable.currentSnapshot().snapshotId());
Review Comment:
What happens to this in case of any empty table? Just a create iceberg table
and nothing. As far as I remember in that case currentSnapshot is null? Just
give a check once.
##
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##
@@ -676,6 +678,32 @@ public void
executeOperation(org.apache.hadoop.hive.ql.metadata.Table hmsTable,
}
}
+ @Override
+ public void createBranchOperation(org.apache.hadoop.hive.ql.metadata.Table
hmsTable,
+ AlterTableCreateBranchSpec createBranchSpec) {
+TableDesc tableDesc = Utilities.getTableDesc(hmsTable);
+Table icebergTable = IcebergTableUtil.getTable(conf,
tableDesc.getProperties());
+
+String branchName = createBranchSpec.getBranchName();
+ManageSnapshots manageSnapshots = icebergTable.manageSnapshots();
+Long snapShotId = Optional.ofNullable(createBranchSpec.getSnapshotId())
+.orElse(icebergTable.currentSnapshot().snapshotId());
+LOG.info("Creating branch {} on iceberg table {}.{}", branchName,
hmsTable.getDbName(),
+hmsTable.getTableName());
+manageSnapshots.createBranch(branchName, snapShotId);
Review Comment:
nit:
snapShotId -> snapshotId
##
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/branch/create/AlterTableCreateBranchAnalyzer.java:
##
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations und
[jira] [Work logged] (HIVE-27304) Exclude CTAS condition while forming storage handler url permissions in HS2 authorizer.
[ https://issues.apache.org/jira/browse/HIVE-27304?focusedWorklogId=859545&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859545 ] ASF GitHub Bot logged work on HIVE-27304: - Author: ASF GitHub Bot Created on: 28/Apr/23 04:23 Start Date: 28/Apr/23 04:23 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4276: URL: https://github.com/apache/hive/pull/4276#issuecomment-1526969904 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4276) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4276&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4276&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4276&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4276&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4276&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4276&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4276&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4276&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4276&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4276&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4276&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4276&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4276&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4276&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 859545) Time Spent: 20m (was: 10m) > Exclude CTAS condition while forming storage handler url permissions in HS2 > authorizer. > --- > > Key: HIVE-27304 > URL: https://issues.apache.org/jira/browse/HIVE-27304 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > CTAS queries with storage handler table as source table doesn't require read > permissions on storage URL for the target table -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27273) Iceberg: Upgrade iceberg to 1.2.1
[
https://issues.apache.org/jira/browse/HIVE-27273?focusedWorklogId=859534&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859534
]
ASF GitHub Bot logged work on HIVE-27273:
-
Author: ASF GitHub Bot
Created on: 28/Apr/23 02:21
Start Date: 28/Apr/23 02:21
Worklog Time Spent: 10m
Work Description: zhangbutao commented on PR #4252:
URL: https://github.com/apache/hive/pull/4252#issuecomment-1526886599
@deniskuzZ Sure， will create tickets to backport these iceberg commits. I
guess @InvisibleProgrammer is woking on that and i am also pleasure to do these
stuff if @InvisibleProgrammer need some assistance.
Thanks.
Issue Time Tracking
---
Worklog Id: (was: 859534)
Time Spent: 2h 50m (was: 2h 40m)
> Iceberg: Upgrade iceberg to 1.2.1
> --
>
> Key: HIVE-27273
> URL: https://issues.apache.org/jira/browse/HIVE-27273
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> [https://iceberg.apache.org/releases/#121-release] Iceberg1.2.1(include
> 1.2.0) has lots of improvement, e.g. _branch commit_ and
> _{{position_deletes}} metadata table._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27163) Column stats are not getting published after an insert query into an external table with custom location
[
https://issues.apache.org/jira/browse/HIVE-27163?focusedWorklogId=859532&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859532
]
ASF GitHub Bot logged work on HIVE-27163:
-
Author: ASF GitHub Bot
Created on: 28/Apr/23 02:06
Start Date: 28/Apr/23 02:06
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4228:
URL: https://github.com/apache/hive/pull/4228#issuecomment-1526876313
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4228)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4228&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4228&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4228&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=CODE_SMELL)
[25 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4228&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4228&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 859532)
Time Spent: 3h 10m (was: 3h)
> Column stats are not getting published after an insert query into an external
> table with custom location
>
>
> Key: HIVE-27163
> URL: https://issues.apache.org/jira/browse/HIVE-27163
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Test case details are below
> *test.q*
> {noformat}
> set hive.stats.column.autogather=true;
> set hive.stats.autogather=true;
> dfs ${system:test.dfs.mkdir} ${system:test.tmp.dir}/test;
> create external table test_custom(age int, name string) stored as orc
> location '/tmp/test';
> insert into test_custom select 1, 'test';
> desc formatted test_custom age;{noformat}
> *test.q.out*
>
>
> {noformat}
> A masked pattern was here
> PREHOOK: type: CREATETA
[jira] [Work logged] (HIVE-26659) TPC-DS query 16, 69, 94 return wrong results.
[
https://issues.apache.org/jira/browse/HIVE-26659?focusedWorklogId=859528&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859528
]
ASF GitHub Bot logged work on HIVE-26659:
-
Author: ASF GitHub Bot
Created on: 28/Apr/23 01:21
Start Date: 28/Apr/23 01:21
Worklog Time Spent: 10m
Work Description: amansinha100 commented on code in PR #4190:
URL: https://github.com/apache/hive/pull/4190#discussion_r1179845226
##
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java:
##
@@ -556,6 +556,9 @@ public void process(Object row, int tag) throws
HiveException {
}
} else {
storage[pos] = emptyList;
+ if (pos != 0 && condn[pos - 1].getType() == JoinDesc.ANTI_JOIN) {
Review Comment:
Just for reference, this check matches the check that is done in
CommonJoinOperator.checkAndGenObject() here:
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java#L973
. Looks good to me.
Issue Time Tracking
---
Worklog Id: (was: 859528)
Time Spent: 2h (was: 1h 50m)
> TPC-DS query 16, 69, 94 return wrong results.
> -
>
> Key: HIVE-26659
> URL: https://issues.apache.org/jira/browse/HIVE-26659
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 4.0.0-alpha-2
>Reporter: Sungwoo Park
>Assignee: Seonggon Namgung
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> TPC-DS query 16, 69, 94 return wrong results when hive.auto.convert.anti.join
> is set to true.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27277) Set up github actions workflow to build and push docker image to docker hub
[ https://issues.apache.org/jira/browse/HIVE-27277?focusedWorklogId=859529&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859529 ] ASF GitHub Bot logged work on HIVE-27277: - Author: ASF GitHub Bot Created on: 28/Apr/23 01:21 Start Date: 28/Apr/23 01:21 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4274: URL: https://github.com/apache/hive/pull/4274#issuecomment-1526848550 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4274) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4274&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4274&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4274&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4274&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4274&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4274&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4274&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4274&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4274&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4274&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4274&resolved=false&types=CODE_SMELL) [16 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4274&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4274&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4274&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 859529) Time Spent: 2h 10m (was: 2h) > Set up github actions workflow to build and push docker image to docker hub > --- > > Key: HIVE-27277 > URL: https://issues.apache.org/jira/browse/HIVE-27277 > Project: Hive > Issue Type: Sub-task >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27304) Exclude CTAS condition while forming storage handler url permissions in HS2 authorizer.
[ https://issues.apache.org/jira/browse/HIVE-27304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27304: -- Labels: pull-request-available (was: ) > Exclude CTAS condition while forming storage handler url permissions in HS2 > authorizer. > --- > > Key: HIVE-27304 > URL: https://issues.apache.org/jira/browse/HIVE-27304 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > CTAS queries with storage handler table as source table doesn't require read > permissions on storage URL for the target table -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27304) Exclude CTAS condition while forming storage handler url permissions in HS2 authorizer.
[ https://issues.apache.org/jira/browse/HIVE-27304?focusedWorklogId=859526&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859526 ] ASF GitHub Bot logged work on HIVE-27304: - Author: ASF GitHub Bot Created on: 28/Apr/23 01:14 Start Date: 28/Apr/23 01:14 Worklog Time Spent: 10m Work Description: saihemanth-cloudera opened a new pull request, #4276: URL: https://github.com/apache/hive/pull/4276 …permissions in HS2 authorizer ### What changes were proposed in this pull request? So that CTAS queries for storage handler-based tables require read permission on storage url. ### Why are the changes needed? So that it doesn't require unnecessary extra permissions. ### Does this PR introduce _any_ user-facing change? Yes, users don't anymore require an extra policy in the authorization service. ### How was this patch tested? Remote cluster Issue Time Tracking --- Worklog Id: (was: 859526) Remaining Estimate: 0h Time Spent: 10m > Exclude CTAS condition while forming storage handler url permissions in HS2 > authorizer. > --- > > Key: HIVE-27304 > URL: https://issues.apache.org/jira/browse/HIVE-27304 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > CTAS queries with storage handler table as source table doesn't require read > permissions on storage URL for the target table -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27294) Remove redundant qt_database_all.q for memory consumption reasons
[
https://issues.apache.org/jira/browse/HIVE-27294?focusedWorklogId=859522&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859522
]
ASF GitHub Bot logged work on HIVE-27294:
-
Author: ASF GitHub Bot
Created on: 28/Apr/23 00:35
Start Date: 28/Apr/23 00:35
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4267:
URL: https://github.com/apache/hive/pull/4267#issuecomment-1526817053
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4267)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4267&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4267&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4267&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=CODE_SMELL)
[16 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4267&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4267&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 859522)
Time Spent: 1h 10m (was: 1h)
> Remove redundant qt_database_all.q for memory consumption reasons
> -
>
> Key: HIVE-27294
> URL: https://issues.apache.org/jira/browse/HIVE-27294
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Currently, while running qt_database_all.q the qtest environment starts and
> runs all the RDMBS docker containers at the same time in beforeTest, which
> might end up in extreme memory consumption. This is suboptimal, and
> considering that the test cases are all covered by single, separate qtests,
> we can simply remove qt_database_all.q.
> {code}
> ./ql/src/test/queries/clientpositive/qt_database_postgres.q
> ./ql/src/test/queries/clientpositive/qt_database_oracle.q
> ./ql/src/test/queries/clientpositive/qt_database_mssql.q
> ./ql/src/test/queries/clientpositive/qt_datab
[jira] [Work logged] (HIVE-27277) Set up github actions workflow to build and push docker image to docker hub
[ https://issues.apache.org/jira/browse/HIVE-27277?focusedWorklogId=859521&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859521 ] ASF GitHub Bot logged work on HIVE-27277: - Author: ASF GitHub Bot Created on: 28/Apr/23 00:06 Start Date: 28/Apr/23 00:06 Worklog Time Spent: 10m Work Description: dengzhhu653 commented on code in PR #4274: URL: https://github.com/apache/hive/pull/4274#discussion_r1179811798 ## .github/workflows/docker-image.yml: ## @@ -0,0 +1,38 @@ +name: ci hive docker image + +on: + push: Review Comment: Once per week makes sense to me for dev, if there are some limitations on the number of the dev images, for example, only keep the latest 10 images for dev. Issue Time Tracking --- Worklog Id: (was: 859521) Time Spent: 2h (was: 1h 50m) > Set up github actions workflow to build and push docker image to docker hub > --- > > Key: HIVE-27277 > URL: https://issues.apache.org/jira/browse/HIVE-27277 > Project: Hive > Issue Type: Sub-task >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27277) Set up github actions workflow to build and push docker image to docker hub
[
https://issues.apache.org/jira/browse/HIVE-27277?focusedWorklogId=859520&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859520
]
ASF GitHub Bot logged work on HIVE-27277:
-
Author: ASF GitHub Bot
Created on: 28/Apr/23 00:03
Start Date: 28/Apr/23 00:03
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #4274:
URL: https://github.com/apache/hive/pull/4274#discussion_r1179810363
##
.github/workflows/docker-image.yml:
##
@@ -0,0 +1,38 @@
+name: ci hive docker image
+
+on:
+ push:
+branches:
+ - "master"
+
+jobs:
+ build:
+runs-on: ubuntu-latest
+steps:
+ -
+name: Checkout
+uses: actions/checkout@v3
+ -
+name: Login to Docker Hub
+uses: docker/login-action@v2
+with:
+ username: ${{ secrets.DOCKERHUB_USERNAME }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ -
+name: Set up Docker Buildx
+uses: docker/setup-buildx-action@v2
+
+ -
+name: Build and push
+uses: docker/build-push-action@v4
+with:
+ context: ./packaging/src/docker/
+ file: ./packaging/src/docker/Dockerfile
+ push: true
+ tags: ${{ secrets.DOCKERHUB_USERNAME }}/hive:test-image
Review Comment:
Can we trigger the build for GA automatically?
https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release
I think it makes some troubles every time we should add the new GA build
manually, it adds extra steps for releasing the new version, sometimes we may
even forget about it.
For the old released version, I think we can push the image manually.
Issue Time Tracking
---
Worklog Id: (was: 859520)
Time Spent: 1h 50m (was: 1h 40m)
> Set up github actions workflow to build and push docker image to docker hub
> ---
>
> Key: HIVE-27277
> URL: https://issues.apache.org/jira/browse/HIVE-27277
> Project: Hive
> Issue Type: Sub-task
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27271) Client connection to HS2 fails when transportMode=http, ssl=true, sslTrustStore specified without trustStorePassword in the JDBC URL
[
https://issues.apache.org/jira/browse/HIVE-27271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng resolved HIVE-27271.
Fix Version/s: 4.0.0
Resolution: Fixed
Merged into master. Thank you [~VenuReddy] for the PR!
> Client connection to HS2 fails when transportMode=http, ssl=true,
> sslTrustStore specified without trustStorePassword in the JDBC URL
>
>
> Key: HIVE-27271
> URL: https://issues.apache.org/jira/browse/HIVE-27271
> Project: Hive
> Issue Type: Bug
>Reporter: Venugopal Reddy K
>Assignee: Venugopal Reddy K
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: image-2023-04-19-14-27-23-665.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> *[Description]*
> Client connection to HS2 fails with transportMode as http, ssl is enabled,
> sslTrustStore is specified without trustStorePassword in the JDBC URL. Where
> as with transportMode as binary, connection is successful without
> trustStorePassword in the connection URL.
> trustStorePassword is not a necessary parameter in connection URL. Connection
> can be established without it.
> From the javadocs
> [Link|https://docs.oracle.com/javase/7/docs/api/java/security/KeyStore.html#load(java.io.InputStream,%20char%5B%5D)]
> A password may be given to unlock the keystore (e.g. the keystore resides on
> a hardware token device), or to check the integrity of the keystore data. If
> a password is not given for integrity checking, then integrity checking is
> not performed.
>
> At present, org.apache.hive.jdbc.HiveConnection#getHttpClient() access
> sslTrustStorePassword null reference and fails as shown below:
> !image-2023-04-19-14-27-23-665.png!
>
> *[Steps to reproduce]*
> {code:java}
> kvenureddy@192 apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin % bin/beeline -u
> "jdbc:hive2://kvrtls-1.kvrtls.root.hwx.site:10001/default;ssl=true;sslTrustStore=/Users/kvenureddy/code/hive/cloudera/hive/packaging/target/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/cm-auto-global_truststore.jks;transportMode=http;httpPath=cliservice;"
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://kvrtls-1.kvrtls.root.hwx.site:10001/default;ssl=true;sslTrustStore=/Users/kvenureddy/code/hive/cloudera/hive/packaging/target/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/cm-auto-global_truststore.jks;transportMode=http;httpPath=cliservice;:
> Could not create an https connection to
> jdbc:hive2://kvrtls-1.kvrtls.root.hwx.site:10001/default;ssl=true;sslTrustStore=/Users/kvenureddy/code/hive/cloudera/hive/packaging/target/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/cm-auto-global_truststore.jks;transportMode=http;httpPath=cliservice;.
> null (state=08S01,code=0)
> kvenureddy@192 apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin %
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27271) Client connection to HS2 fails when transportMode=http, ssl=true, sslTrustStore specified without trustStorePassword in the JDBC URL
[
https://issues.apache.org/jira/browse/HIVE-27271?focusedWorklogId=859519&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859519
]
ASF GitHub Bot logged work on HIVE-27271:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 23:21
Start Date: 27/Apr/23 23:21
Worklog Time Spent: 10m
Work Description: dengzhhu653 merged PR #4262:
URL: https://github.com/apache/hive/pull/4262
Issue Time Tracking
---
Worklog Id: (was: 859519)
Time Spent: 40m (was: 0.5h)
> Client connection to HS2 fails when transportMode=http, ssl=true,
> sslTrustStore specified without trustStorePassword in the JDBC URL
>
>
> Key: HIVE-27271
> URL: https://issues.apache.org/jira/browse/HIVE-27271
> Project: Hive
> Issue Type: Bug
>Reporter: Venugopal Reddy K
>Assignee: Venugopal Reddy K
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-04-19-14-27-23-665.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> *[Description]*
> Client connection to HS2 fails with transportMode as http, ssl is enabled,
> sslTrustStore is specified without trustStorePassword in the JDBC URL. Where
> as with transportMode as binary, connection is successful without
> trustStorePassword in the connection URL.
> trustStorePassword is not a necessary parameter in connection URL. Connection
> can be established without it.
> From the javadocs
> [Link|https://docs.oracle.com/javase/7/docs/api/java/security/KeyStore.html#load(java.io.InputStream,%20char%5B%5D)]
> A password may be given to unlock the keystore (e.g. the keystore resides on
> a hardware token device), or to check the integrity of the keystore data. If
> a password is not given for integrity checking, then integrity checking is
> not performed.
>
> At present, org.apache.hive.jdbc.HiveConnection#getHttpClient() access
> sslTrustStorePassword null reference and fails as shown below:
> !image-2023-04-19-14-27-23-665.png!
>
> *[Steps to reproduce]*
> {code:java}
> kvenureddy@192 apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin % bin/beeline -u
> "jdbc:hive2://kvrtls-1.kvrtls.root.hwx.site:10001/default;ssl=true;sslTrustStore=/Users/kvenureddy/code/hive/cloudera/hive/packaging/target/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/cm-auto-global_truststore.jks;transportMode=http;httpPath=cliservice;"
> Error: Could not open client transport with JDBC Uri:
> jdbc:hive2://kvrtls-1.kvrtls.root.hwx.site:10001/default;ssl=true;sslTrustStore=/Users/kvenureddy/code/hive/cloudera/hive/packaging/target/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/cm-auto-global_truststore.jks;transportMode=http;httpPath=cliservice;:
> Could not create an https connection to
> jdbc:hive2://kvrtls-1.kvrtls.root.hwx.site:10001/default;ssl=true;sslTrustStore=/Users/kvenureddy/code/hive/cloudera/hive/packaging/target/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin/cm-auto-global_truststore.jks;transportMode=http;httpPath=cliservice;.
> null (state=08S01,code=0)
> kvenureddy@192 apache-hive-3.1.3000.2023.0.15.0-SNAPSHOT-bin %
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27163) Column stats are not getting published after an insert query into an external table with custom location
[
https://issues.apache.org/jira/browse/HIVE-27163?focusedWorklogId=859517&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859517
]
ASF GitHub Bot logged work on HIVE-27163:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 23:18
Start Date: 27/Apr/23 23:18
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #4228:
URL: https://github.com/apache/hive/pull/4228#discussion_r1179790929
##
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##
@@ -339,17 +339,16 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_ice_puffin
col_name a
data_type int
-min1
Review Comment:
@simhadri-g, seems like the `desc formatted tbl_ice_puffin a` gets status
from metastore though `hive.iceberg.stats.source=iceberg`, cloud you please
check?
Issue Time Tracking
---
Worklog Id: (was: 859517)
Time Spent: 2h 50m (was: 2h 40m)
> Column stats are not getting published after an insert query into an external
> table with custom location
>
>
> Key: HIVE-27163
> URL: https://issues.apache.org/jira/browse/HIVE-27163
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Test case details are below
> *test.q*
> {noformat}
> set hive.stats.column.autogather=true;
> set hive.stats.autogather=true;
> dfs ${system:test.dfs.mkdir} ${system:test.tmp.dir}/test;
> create external table test_custom(age int, name string) stored as orc
> location '/tmp/test';
> insert into test_custom select 1, 'test';
> desc formatted test_custom age;{noformat}
> *test.q.out*
>
>
> {noformat}
> A masked pattern was here
> PREHOOK: type: CREATETABLE
> A masked pattern was here
> PREHOOK: Output: database:default
> PREHOOK: Output: default@test_custom
> A masked pattern was here
> POSTHOOK: type: CREATETABLE
> A masked pattern was here
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@test_custom
> PREHOOK: query: insert into test_custom select 1, 'test'
> PREHOOK: type: QUERY
> PREHOOK: Input: _dummy_database@_dummy_table
> PREHOOK: Output: default@test_custom
> POSTHOOK: query: insert into test_custom select 1, 'test'
> POSTHOOK: type: QUERY
> POSTHOOK: Input: _dummy_database@_dummy_table
> POSTHOOK: Output: default@test_custom
> POSTHOOK: Lineage: test_custom.age SIMPLE []
> POSTHOOK: Lineage: test_custom.name SIMPLE []
> PREHOOK: query: desc formatted test_custom age
> PREHOOK: type: DESCTABLE
> PREHOOK: Input: default@test_custom
> POSTHOOK: query: desc formatted test_custom age
> POSTHOOK: type: DESCTABLE
> POSTHOOK: Input: default@test_custom
> col_name age
> data_type int
> min
> max
> num_nulls
> distinct_count
> avg_col_len
> max_col_len
> num_trues
> num_falses
> bit_vector
> comment from deserializer{noformat}
> As we can see from desc formatted output, column stats were not populated
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27163) Column stats are not getting published after an insert query into an external table with custom location
[
https://issues.apache.org/jira/browse/HIVE-27163?focusedWorklogId=859518&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859518
]
ASF GitHub Bot logged work on HIVE-27163:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 23:18
Start Date: 27/Apr/23 23:18
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #4228:
URL: https://github.com/apache/hive/pull/4228#discussion_r1179790929
##
iceberg/iceberg-handler/src/test/results/positive/col_stats.q.out:
##
@@ -339,17 +339,16 @@ POSTHOOK: type: DESCTABLE
POSTHOOK: Input: default@tbl_ice_puffin
col_name a
data_type int
-min1
Review Comment:
Hi @simhadri-g, seems like the `desc formatted tbl_ice_puffin a` gets status
from metastore though `hive.iceberg.stats.source=iceberg`, cloud you please
check?
Issue Time Tracking
---
Worklog Id: (was: 859518)
Time Spent: 3h (was: 2h 50m)
> Column stats are not getting published after an insert query into an external
> table with custom location
>
>
> Key: HIVE-27163
> URL: https://issues.apache.org/jira/browse/HIVE-27163
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Test case details are below
> *test.q*
> {noformat}
> set hive.stats.column.autogather=true;
> set hive.stats.autogather=true;
> dfs ${system:test.dfs.mkdir} ${system:test.tmp.dir}/test;
> create external table test_custom(age int, name string) stored as orc
> location '/tmp/test';
> insert into test_custom select 1, 'test';
> desc formatted test_custom age;{noformat}
> *test.q.out*
>
>
> {noformat}
> A masked pattern was here
> PREHOOK: type: CREATETABLE
> A masked pattern was here
> PREHOOK: Output: database:default
> PREHOOK: Output: default@test_custom
> A masked pattern was here
> POSTHOOK: type: CREATETABLE
> A masked pattern was here
> POSTHOOK: Output: database:default
> POSTHOOK: Output: default@test_custom
> PREHOOK: query: insert into test_custom select 1, 'test'
> PREHOOK: type: QUERY
> PREHOOK: Input: _dummy_database@_dummy_table
> PREHOOK: Output: default@test_custom
> POSTHOOK: query: insert into test_custom select 1, 'test'
> POSTHOOK: type: QUERY
> POSTHOOK: Input: _dummy_database@_dummy_table
> POSTHOOK: Output: default@test_custom
> POSTHOOK: Lineage: test_custom.age SIMPLE []
> POSTHOOK: Lineage: test_custom.name SIMPLE []
> PREHOOK: query: desc formatted test_custom age
> PREHOOK: type: DESCTABLE
> PREHOOK: Input: default@test_custom
> POSTHOOK: query: desc formatted test_custom age
> POSTHOOK: type: DESCTABLE
> POSTHOOK: Input: default@test_custom
> col_name age
> data_type int
> min
> max
> num_nulls
> distinct_count
> avg_col_len
> max_col_len
> num_trues
> num_falses
> bit_vector
> comment from deserializer{noformat}
> As we can see from desc formatted output, column stats were not populated
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27296) HiveRelDecorrelator does not handle correlation with Values
[
https://issues.apache.org/jira/browse/HIVE-27296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-27296:
--
Labels: pull-request-available (was: )
> HiveRelDecorrelator does not handle correlation with Values
> ---
>
> Key: HIVE-27296
> URL: https://issues.apache.org/jira/browse/HIVE-27296
> Project: Hive
> Issue Type: Bug
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{HiveRelDecorrelator}} does not cope well with {{Values}} expressions
> and when such expression exists in the plan it fails to remove the respective
> {{{}Correlate{}}}.
> In HIVE-27278, we discovered a query that has a correlation over an empty
> {{Values}} expression.
> {code:sql}
> EXPLAIN CBO SELECT id FROM t1 WHERE NULL IN (SELECT NULL FROM t2 where t1.id
> = t2.id);{code}
> The CBO plan after decorrelation is shown below.
> {noformat}
> HiveProject(id=[$0])
> LogicalCorrelate(correlation=[$cor0], joinType=[semi], requiredColumns=[{}])
> HiveTableScan(table=[[default, t1]], table:alias=[t1])
> HiveValues(tuples=[[]])
> {noformat}
> Although, in HIVE-27278 we could find a solution for a plan that contains an
> empty {{Values}} there can be queries with correlations on non-empty
> {{Values}} and for those we don't have a solution at the moment.
> Normally after decorrelation we shouldn't have any {{Correlate}} expressions
> in the plan.
> The problem starts from
> [HiveRelDecorrelator.decorrelate(Values)|https://github.com/apache/hive/blob/59058c65457fb7ab9d8575a555034e6633962661/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java#L471]
> that returns null when it encounters the {{Values}} expression.
> Later, in
> [HiveRelDecorrelator.decorrelate(Correlate)|https://github.com/apache/hive/blob/59058c65457fb7ab9d8575a555034e6633962661/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java#L1247]
> it will bail out when treating the {{Correlate}} since one of the inputs is
> not rewritten.
> The problem is still there in latest Calcite (CALCITE-5568).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27296) HiveRelDecorrelator does not handle correlation with Values
[
https://issues.apache.org/jira/browse/HIVE-27296?focusedWorklogId=859516&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859516
]
ASF GitHub Bot logged work on HIVE-27296:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 22:43
Start Date: 27/Apr/23 22:43
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4275:
URL: https://github.com/apache/hive/pull/4275#issuecomment-1526664111
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4275)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4275&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4275&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4275&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4275&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4275&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4275&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4275&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4275&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4275&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4275&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4275&resolved=false&types=CODE_SMELL)
[16 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4275&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4275&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4275&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 859516)
Remaining Estimate: 0h
Time Spent: 10m
> HiveRelDecorrelator does not handle correlation with Values
> ---
>
> Key: HIVE-27296
> URL: https://issues.apache.org/jira/browse/HIVE-27296
> Project: Hive
> Issue Type: Bug
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The {{HiveRelDecorrelator}} does not cope well with {{Values}} expressions
> and when such expression exists in the plan it fails to remove the respective
> {{{}Correlate{}}}.
> In HIVE-27278, we discovered a query that has a correlation over an empty
> {{Values}} expression.
> {code:sql}
> EXPLAIN CBO SELECT id FROM t1 WHERE NULL IN (SELECT NULL FROM t2 where t1.id
> = t2.id);{code}
> The CBO plan after decorrelation is shown below.
> {noformat}
> HiveProject(id=[$0])
> LogicalCorrelate(correlation=[$cor0], joinType=[semi], requiredColumns=[{}])
> HiveTableScan(table=[[default, t1]], table:al
[jira] [Resolved] (HIVE-26838) Add a new event to improve cache performance in external systems that communicates with HMS.
[ https://issues.apache.org/jira/browse/HIVE-26838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala resolved HIVE-26838. -- Fix Version/s: 4.0.0 Resolution: Fixed > Add a new event to improve cache performance in external systems that > communicates with HMS. > > > Key: HIVE-26838 > URL: https://issues.apache.org/jira/browse/HIVE-26838 > Project: Hive > Issue Type: New Feature > Components: Hive, Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 3h > Remaining Estimate: 0h > > Adding support for a new event "Reload event" in the HMS (HiveMetaStore). > This event can be used by external services that depend on HMS for metadata > operations to improve its cache performance. In the distributed environment > where there are replicas of an external service (with its own cache in each > of these replicas) talking to HMS for metadata operations, the reload event > can be used to address the cache performance and ensure consistency among all > the replicas for a given table/partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26838) Add a new event to improve cache performance in external systems that communicates with HMS.
[ https://issues.apache.org/jira/browse/HIVE-26838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717390#comment-17717390 ] Sai Hemanth Gantasala commented on HIVE-26838: -- Patch merged to master. [~dengzh] thanks for the review. > Add a new event to improve cache performance in external systems that > communicates with HMS. > > > Key: HIVE-26838 > URL: https://issues.apache.org/jira/browse/HIVE-26838 > Project: Hive > Issue Type: New Feature > Components: Hive, Standalone Metastore >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Adding support for a new event "Reload event" in the HMS (HiveMetaStore). > This event can be used by external services that depend on HMS for metadata > operations to improve its cache performance. In the distributed environment > where there are replicas of an external service (with its own cache in each > of these replicas) talking to HMS for metadata operations, the reload event > can be used to address the cache performance and ensure consistency among all > the replicas for a given table/partition. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27304) Exclude CTAS condition while forming storage handler url permissions in HS2 authorizer.
Sai Hemanth Gantasala created HIVE-27304: Summary: Exclude CTAS condition while forming storage handler url permissions in HS2 authorizer. Key: HIVE-27304 URL: https://issues.apache.org/jira/browse/HIVE-27304 Project: Hive Issue Type: Bug Components: Hive Reporter: Sai Hemanth Gantasala Assignee: Sai Hemanth Gantasala CTAS queries with storage handler table as source table doesn't require read permissions on storage URL for the target table -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-23394) TestJdbcGenericUDTFGetSplits2#testGenericUDTFOrderBySplitCount1 is flaky
[
https://issues.apache.org/jira/browse/HIVE-23394?focusedWorklogId=859477&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859477
]
ASF GitHub Bot logged work on HIVE-23394:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 20:37
Start Date: 27/Apr/23 20:37
Worklog Time Spent: 10m
Work Description: simhadri-g commented on PR #4249:
URL: https://github.com/apache/hive/pull/4249#issuecomment-1526404188
Hi Guys,
The leaking thread has been fixed by this PR.
https://github.com/apache/hive/pull/2967
The test we are enabling checks the following 2 udtfs:
"get_splits()"
"get_llap_splits()".
Both the tests have similar code and I have merged it into a single file to
reduce code duplication.
Issue Time Tracking
---
Worklog Id: (was: 859477)
Time Spent: 1h 40m (was: 1.5h)
> TestJdbcGenericUDTFGetSplits2#testGenericUDTFOrderBySplitCount1 is flaky
>
>
> Key: HIVE-23394
> URL: https://issues.apache.org/jira/browse/HIVE-23394
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> both
> TestJdbcGenericUDTFGetSplits2.testGenericUDTFOrderBySplitCount1 and
> TestJdbcGenericUDTFGetSplits.testGenericUDTFOrderBySplitCount1
> can fail with the exception below
> seems like the connection was lost
> {code}
> Error Message
> Failed to close statement
> Stacktrace
> java.sql.SQLException: Failed to close statement
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:200)
> at
> org.apache.hive.jdbc.HiveStatement.closeClientOperation(HiveStatement.java:205)
> at org.apache.hive.jdbc.HiveStatement.close(HiveStatement.java:222)
> at
> org.apache.hive.jdbc.AbstractTestJdbcGenericUDTFGetSplits.runQuery(AbstractTestJdbcGenericUDTFGetSplits.java:135)
> at
> org.apache.hive.jdbc.AbstractTestJdbcGenericUDTFGetSplits.testGenericUDTFOrderBySplitCount1(AbstractTestJdbcGenericUDTFGetSplits.java:164)
> at
> org.apache.hive.jdbc.TestJdbcGenericUDTFGetSplits2.testGenericUDTFOrderBySplitCount1(TestJdbcGenericUDTFGetSplits2.java:28)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: org.apache.thrift.TApplicationException: CloseOperation failed:
> out of sequence response
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:84)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_CloseOperation(TCLIService.java:521)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:508)
> at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1732)
> at com.sun.proxy.$Proxy146.CloseOperation(Unknown Source)
> at
> org.apache.hive.jdbc.HiveStatement.closeStatementIfNeeded(HiveStatement.java:193)
> ... 14 more
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26827) Add configs in error message to workaround predicate issue with Parquet on TIMESTAMP data type
[
https://issues.apache.org/jira/browse/HIVE-26827?focusedWorklogId=859476&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859476
]
ASF GitHub Bot logged work on HIVE-26827:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 20:25
Start Date: 27/Apr/23 20:25
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4200:
URL: https://github.com/apache/hive/pull/4200#issuecomment-1526377991
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4200)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4200&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4200&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4200&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4200&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4200&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4200&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4200&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4200&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4200&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4200&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4200&resolved=false&types=CODE_SMELL)
[16 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4200&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4200&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4200&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 859476)
Time Spent: 1h 10m (was: 1h)
> Add configs in error message to workaround predicate issue with Parquet on
> TIMESTAMP data type
> --
>
> Key: HIVE-26827
> URL: https://issues.apache.org/jira/browse/HIVE-26827
> Project: Hive
> Issue Type: Improvement
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The below query fails with error
> {noformat}
> select * from db.parquet_table_with_timestamp where created_date_utc between
> '2022-11-05 00:01:01' and '2022-11-08 23:59:59'{noformat}
>
> We can workaround the issue below
>
> {noformat}
> 2022-11-10 06:43:36,751 [ERROR] [TezChild]
> |read.ParquetFilterPredicateConverter|: fail to build predicate filter leaf
> with errors org.apache.hadoop.hive.ql.metadata.HiveException: Conversion to
> Parquet FilterPredicate not supported for TIMES
[jira] [Work logged] (HIVE-27285) Add TableMeta ownership for filterTableMetas
[
https://issues.apache.org/jira/browse/HIVE-27285?focusedWorklogId=859471&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859471
]
ASF GitHub Bot logged work on HIVE-27285:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 19:53
Start Date: 27/Apr/23 19:53
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4258:
URL: https://github.com/apache/hive/pull/4258#issuecomment-1526303303
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4258)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4258&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4258&resolved=false&types=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4258&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4258&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4258&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4258&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4258&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4258&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4258&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4258&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4258&resolved=false&types=CODE_SMELL)
[32 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4258&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4258&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4258&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 859471)
Time Spent: 3h 20m (was: 3h 10m)
> Add TableMeta ownership for filterTableMetas
>
>
> Key: HIVE-27285
> URL: https://issues.apache.org/jira/browse/HIVE-27285
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Currently TableMeta does not include ownership information which makes it
> difficult for filterTableMetas to efficiently filter based on ${OWNER}
> privileges.
> We should add ownership information to TableMeta and utilizing it in
> filterTableMetas authorization checks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27281) Add ability of masking to Beeline q-tests
[
https://issues.apache.org/jira/browse/HIVE-27281?focusedWorklogId=859468&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859468
]
ASF GitHub Bot logged work on HIVE-27281:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 19:37
Start Date: 27/Apr/23 19:37
Worklog Time Spent: 10m
Work Description: difin commented on code in PR #4254:
URL: https://github.com/apache/hive/pull/4254#discussion_r1179603252
##
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java:
##
@@ -97,6 +101,8 @@ public CoreBeeLineDriver(AbstractCliConfig testCliConfig) {
initScript = new File(testScriptDirectory,
testCliConfig.getInitScript());
}
cleanupScript = new File(testScriptDirectory,
testCliConfig.getCleanupScript());
+this.replaceHandler = new QTestReplaceHandler();
Review Comment:
As I understand form the code, QTestUtil can't be used as is because it
doesn't execute commands using beeline.
After digging into other drivers that use QTestUtil, I saw that QTestUtil
executes commands from q-files using ReExecDriver.java which lives in hive-ql
module, while CoreBeeLineDriver executes commands from q-files using
Beeline.java.
Issue Time Tracking
---
Worklog Id: (was: 859468)
Time Spent: 1h 20m (was: 1h 10m)
> Add ability of masking to Beeline q-tests
> -
>
> Key: HIVE-27281
> URL: https://issues.apache.org/jira/browse/HIVE-27281
> Project: Hive
> Issue Type: Improvement
>Reporter: Dmitriy Fingerman
>Assignee: Dmitriy Fingerman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26659) TPC-DS query 16, 69, 94 return wrong results.
[ https://issues.apache.org/jira/browse/HIVE-26659?focusedWorklogId=859467&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859467 ] ASF GitHub Bot logged work on HIVE-26659: - Author: ASF GitHub Bot Created on: 27/Apr/23 18:34 Start Date: 27/Apr/23 18:34 Worklog Time Spent: 10m Work Description: scarlin-cloudera commented on code in PR #4190: URL: https://github.com/apache/hive/pull/4190#discussion_r1179548225 ## ql/src/test/queries/clientpositive/antijoin2.q: ## @@ -0,0 +1,75 @@ +set hive.merge.nway.joins=false; +set hive.vectorized.execution.enabled=false; +set hive.auto.convert.join=true; +set hive.auto.convert.anti.join=true; + +drop table if exists tt1; +drop table if exists tt2; +drop table if exists tt3; + +create table tt1 (ws_order_number bigint, ws_ext_ship_cost decimal(7, 2)); +create table tt2 (ws_order_number bigint); +create table tt3 (wr_order_number bigint); + +insert into tt1 values (42, 3093.96), (1041, 299.28), (1378, 85.56), (1378, 719.44), (1395, 145.68); +insert into tt2 values (1378), (1395); +insert into tt3 values (42), (1041); + + Issue Time Tracking --- Worklog Id: (was: 859467) Time Spent: 1h 50m (was: 1h 40m) > TPC-DS query 16, 69, 94 return wrong results. > - > > Key: HIVE-26659 > URL: https://issues.apache.org/jira/browse/HIVE-26659 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0-alpha-2 >Reporter: Sungwoo Park >Assignee: Seonggon Namgung >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > TPC-DS query 16, 69, 94 return wrong results when hive.auto.convert.anti.join > is set to true. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27247) Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue and HIVE-19662: Upgrade Avro to 1.8.2
[ https://issues.apache.org/jira/browse/HIVE-27247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-27247: Summary: Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue and HIVE-19662: Upgrade Avro to 1.8.2 (was: Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue) > Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue and > HIVE-19662: Upgrade Avro to 1.8.2 > - > > Key: HIVE-27247 > URL: https://issues.apache.org/jira/browse/HIVE-27247 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue > and HIVE-19662: Upgrade Avro to 1.8.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27247) Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue
[ https://issues.apache.org/jira/browse/HIVE-27247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-27247: Description: Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue and HIVE-19662: Upgrade Avro to 1.8.2 was:Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue > Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue > --- > > Key: HIVE-27247 > URL: https://issues.apache.org/jira/browse/HIVE-27247 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue > and HIVE-19662: Upgrade Avro to 1.8.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27247) Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue
[ https://issues.apache.org/jira/browse/HIVE-27247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27247. - Fix Version/s: 3.2.0 Resolution: Fixed Thanks [~diksha193] for the contribution! > Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue > --- > > Key: HIVE-27247 > URL: https://issues.apache.org/jira/browse/HIVE-27247 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27247) Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue
[ https://issues.apache.org/jira/browse/HIVE-27247?focusedWorklogId=859456&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859456 ] ASF GitHub Bot logged work on HIVE-27247: - Author: ASF GitHub Bot Created on: 27/Apr/23 17:25 Start Date: 27/Apr/23 17:25 Worklog Time Spent: 10m Work Description: sankarh merged PR #4218: URL: https://github.com/apache/hive/pull/4218 Issue Time Tracking --- Worklog Id: (was: 859456) Time Spent: 40m (was: 0.5h) > Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue > --- > > Key: HIVE-27247 > URL: https://issues.apache.org/jira/browse/HIVE-27247 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27247) Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue
[ https://issues.apache.org/jira/browse/HIVE-27247?focusedWorklogId=859452&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859452 ] ASF GitHub Bot logged work on HIVE-27247: - Author: ASF GitHub Bot Created on: 27/Apr/23 17:18 Start Date: 27/Apr/23 17:18 Worklog Time Spent: 10m Work Description: sankarh closed pull request #4217: HIVE-27247: Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue URL: https://github.com/apache/hive/pull/4217 Issue Time Tracking --- Worklog Id: (was: 859452) Time Spent: 0.5h (was: 20m) > Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue > --- > > Key: HIVE-27247 > URL: https://issues.apache.org/jira/browse/HIVE-27247 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Backport of HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27248) Backport of HIVE-24653: Race condition between compactor marker generation and get splits
[ https://issues.apache.org/jira/browse/HIVE-27248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27248. - Fix Version/s: 3.2.0 Resolution: Fixed Thanks [~diksha193] for the contribution! > Backport of HIVE-24653: Race condition between compactor marker generation > and get splits > - > > Key: HIVE-27248 > URL: https://issues.apache.org/jira/browse/HIVE-27248 > Project: Hive > Issue Type: Sub-task >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Backport of HIVE-24653: Race condition between compactor marker generation > and get splits -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27248) Backport of HIVE-24653: Race condition between compactor marker generation and get splits
[ https://issues.apache.org/jira/browse/HIVE-27248?focusedWorklogId=859447&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859447 ] ASF GitHub Bot logged work on HIVE-27248: - Author: ASF GitHub Bot Created on: 27/Apr/23 17:04 Start Date: 27/Apr/23 17:04 Worklog Time Spent: 10m Work Description: sankarh merged PR #4219: URL: https://github.com/apache/hive/pull/4219 Issue Time Tracking --- Worklog Id: (was: 859447) Time Spent: 0.5h (was: 20m) > Backport of HIVE-24653: Race condition between compactor marker generation > and get splits > - > > Key: HIVE-27248 > URL: https://issues.apache.org/jira/browse/HIVE-27248 > Project: Hive > Issue Type: Sub-task >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Backport of HIVE-24653: Race condition between compactor marker generation > and get splits -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27058) Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 into branch-3
[ https://issues.apache.org/jira/browse/HIVE-27058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-27058: Summary: Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 into branch-3 (was: Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1) > Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 into branch-3 > - > > Key: HIVE-27058 > URL: https://issues.apache.org/jira/browse/HIVE-27058 > Project: Hive > Issue Type: Sub-task >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: hive-3.2.0-must, pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27058) Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 into branch-3
[ https://issues.apache.org/jira/browse/HIVE-27058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27058. - Fix Version/s: 3.2.0 Resolution: Fixed Thanks [~diksha193] for the contribution! > Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 into branch-3 > - > > Key: HIVE-27058 > URL: https://issues.apache.org/jira/browse/HIVE-27058 > Project: Hive > Issue Type: Sub-task >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: hive-3.2.0-must, pull-request-available > Fix For: 3.2.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27058) Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1
[ https://issues.apache.org/jira/browse/HIVE-27058?focusedWorklogId=859441&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859441 ] ASF GitHub Bot logged work on HIVE-27058: - Author: ASF GitHub Bot Created on: 27/Apr/23 16:58 Start Date: 27/Apr/23 16:58 Worklog Time Spent: 10m Work Description: sankarh merged PR #4192: URL: https://github.com/apache/hive/pull/4192 Issue Time Tracking --- Worklog Id: (was: 859441) Time Spent: 0.5h (was: 20m) > Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1 > - > > Key: HIVE-27058 > URL: https://issues.apache.org/jira/browse/HIVE-27058 > Project: Hive > Issue Type: Sub-task >Reporter: Diksha >Assignee: Diksha >Priority: Major > Labels: hive-3.2.0-must, pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Backport of HIVE-24316: Upgrade ORC from 1.5.6 to 1.5.8 in branch-3.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27186) A persistent property store
[ https://issues.apache.org/jira/browse/HIVE-27186?focusedWorklogId=859432&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859432 ] ASF GitHub Bot logged work on HIVE-27186: - Author: ASF GitHub Bot Created on: 27/Apr/23 16:22 Start Date: 27/Apr/23 16:22 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4194: URL: https://github.com/apache/hive/pull/4194#issuecomment-1525991626 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4194) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=BUG) [4 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4194&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4194&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4194&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=CODE_SMELL) [112 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4194&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4194&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 859432) Time Spent: 14h 20m (was: 14h 10m) > A persistent property store > > > Key: HIVE-27186 > URL: https://issues.apache.org/jira/browse/HIVE-27186 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0-alpha-2 >Reporter: Henri Biestro >Assignee: Henri Biestro >Priority: Major > Labels: pull-request-available > Time Spent: 14h 20m > Remaining Estimate: 0h > > WHAT > A persistent property store usable as a support facility for any metadata > augmentation feature. > WHY > When adding new meta-data oriented features, we usually need to persist > information linking the feature data and the HiveMetaStore objects it applies > to. Any information related to a database, a table or the cluster - like > statistics for example or any operational data state or data (think rolling > backup) - fall in this use-case. > Typically, accommodating such a feature requires modifying the Metastore > database schema by adding or altering a ta
[jira] [Work logged] (HIVE-26982) Select * from a table containing timestamp column with default defined using TIMESTAMPLOCALTZ fails with error " ORC doesn't handle primitive category TIMESTAMPLOCALT
[
https://issues.apache.org/jira/browse/HIVE-26982?focusedWorklogId=859431&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859431
]
ASF GitHub Bot logged work on HIVE-26982:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 15:59
Start Date: 27/Apr/23 15:59
Worklog Time Spent: 10m
Work Description: aturoczy commented on code in PR #4265:
URL: https://github.com/apache/hive/pull/4265#discussion_r1179387584
##
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java:
##
@@ -220,6 +220,9 @@ public static List
setTypeBuilderFromSchema(
case TIMESTAMP:
type.setKind(OrcProto.Type.Kind.TIMESTAMP);
break;
+case TIMESTAMP_INSTANT:
Review Comment:
Code is easy and seems legit. Just could you please write a test to cover
the timestamp?
Issue Time Tracking
---
Worklog Id: (was: 859431)
Time Spent: 40m (was: 0.5h)
> Select * from a table containing timestamp column with default defined using
> TIMESTAMPLOCALTZ fails with error " ORC doesn't handle primitive category
> TIMESTAMPLOCALTZ"
>
>
> Key: HIVE-26982
> URL: https://issues.apache.org/jira/browse/HIVE-26982
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Reporter: Dharmik Thakkar
>Assignee: Zoltán Rátkai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Select * from a table containing timestamp column with default defined using
> TIMESTAMPLOCALTZ fails with error " ORC doesn't handle primitive category
> TIMESTAMPLOCALTZ"
> Logs
> {code:java}
> 2023-01-24T20:37:48,831 INFO [pool-2-thread-1] jdbc.TestDriver: Beginning
> Test at 2023-01-24 20:37:48,831
> 2023-01-24T20:37:48,833 INFO [pool-2-thread-1] jdbc.TestDriver: BEGIN MAIN
> 2023-01-24T20:37:48,834 INFO [pool-9-thread-1] jdbc.TestDriver: Running
> SessionGroup{name='SG_JZSL3SA0OG', initialDelay=0, repeats=1, repeatDelay=0}
> 2023-01-24T20:37:48,834 INFO [pool-9-thread-1] jdbc.TestDriver: Connecting
> as user 'hrt_qa'
> 2023-01-24T20:37:49,173 INFO [pool-9-thread-1] jdbc.TestDriver: Query: drop
> table if exists t1_default
> 2023-01-24T20:37:49,237 INFO [Thread-64] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230124203749_09b0f95f-4cf1-4c2c-9f08-1b91fdb4a6ca):
> drop table if exists t1_default
> 2023-01-24T20:37:49,299 INFO [Thread-64] jdbc.TestDriver: INFO : Semantic
> Analysis Completed (retrial = false)
> 2023-01-24T20:37:49,299 INFO [Thread-64] jdbc.TestDriver: INFO : Created
> Hive schema: Schema(fieldSchemas:null, properties:null)
> 2023-01-24T20:37:49,299 INFO [Thread-64] jdbc.TestDriver: INFO : Completed
> compiling
> command(queryId=hive_20230124203749_09b0f95f-4cf1-4c2c-9f08-1b91fdb4a6ca);
> Time taken: 0.031 seconds
> 2023-01-24T20:37:49,299 INFO [Thread-64] jdbc.TestDriver: INFO : Executing
> command(queryId=hive_20230124203749_09b0f95f-4cf1-4c2c-9f08-1b91fdb4a6ca):
> drop table if exists t1_default
> 2023-01-24T20:37:49,299 INFO [Thread-64] jdbc.TestDriver: INFO : Starting
> task [Stage-0:DDL] in serial mode
> 2023-01-24T20:37:49,299 INFO [Thread-64] jdbc.TestDriver: INFO : Completed
> executing
> command(queryId=hive_20230124203749_09b0f95f-4cf1-4c2c-9f08-1b91fdb4a6ca);
> Time taken: 0.012 seconds
> 2023-01-24T20:37:49,299 INFO [Thread-64] jdbc.TestDriver: INFO : OK
> 2023-01-24T20:37:49,416 INFO [pool-9-thread-1] jdbc.TestDriver: No output to
> verify
> 2023-01-24T20:37:49,416 INFO [pool-9-thread-1] jdbc.TestDriver: Query:
> create table t1_default ( t tinyint default 1Y, si smallint default 1S,
> i int default 1,b bigint default 1L, f double default
> double(5.7), d double, s varchar(25) default cast('col1' as
> varchar(25)), dc decimal(38,18), bo varchar(5), v varchar(25),
> c char(25) default cast('var1' as char(25)), ts timestamp DEFAULT
> TIMESTAMP'2016-02-22 12:45:07.0', dt date default
> cast('2015-03-12' as DATE), tz timestamp with local time zone DEFAULT
> TIMESTAMPLOCALTZ'2016-01-03 12:26:34 America/Los_Angeles') STORED AS
> TEXTFILE
> 2023-01-24T20:37:49,476 INFO [Thread-65] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230124203749_75ffcf31-6bd6-46d7-ba02-f39efb2c4279):
> create table t1_default ( t tinyint default 1Y, si smallint default 1S,
> i int default 1,b bigint default 1L, f double default
> double(5.7), d double, s varchar(25) default cast('col1' as
> varchar(25)), dc decimal(3
[jira] [Work logged] (HIVE-27163) Column stats are not getting published after an insert query into an external table with custom location
[
https://issues.apache.org/jira/browse/HIVE-27163?focusedWorklogId=859427&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859427
]
ASF GitHub Bot logged work on HIVE-27163:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 15:23
Start Date: 27/Apr/23 15:23
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4228:
URL: https://github.com/apache/hive/pull/4228#issuecomment-1525894367
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4228)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4228&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4228&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4228&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=CODE_SMELL)
[25 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4228&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4228&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 859427)
Time Spent: 2h 40m (was: 2.5h)
> Column stats are not getting published after an insert query into an external
> table with custom location
>
>
> Key: HIVE-27163
> URL: https://issues.apache.org/jira/browse/HIVE-27163
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Test case details are below
> *test.q*
> {noformat}
> set hive.stats.column.autogather=true;
> set hive.stats.autogather=true;
> dfs ${system:test.dfs.mkdir} ${system:test.tmp.dir}/test;
> create external table test_custom(age int, name string) stored as orc
> location '/tmp/test';
> insert into test_custom select 1, 'test';
> desc formatted test_custom age;{noformat}
> *test.q.out*
>
>
> {noformat}
> A masked pattern was here
> PREHOOK: type: CREATE
[jira] [Work logged] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?focusedWorklogId=859425&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859425
]
ASF GitHub Bot logged work on HIVE-27172:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 15:13
Start Date: 27/Apr/23 15:13
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4150:
URL: https://github.com/apache/hive/pull/4150#discussion_r1179317009
##
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTimeout.java:
##
@@ -117,7 +123,7 @@ public void testResetTimeout() throws Exception {
.build(conf);
try {
client.createDatabase(db);
-} catch (MetaException e) {
+} catch (Exception e) {
Review Comment:
why is this change?
Issue Time Tracking
---
Worklog Id: (was: 859425)
Time Spent: 2.5h (was: 2h 20m)
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27172) Add the HMS client connection timeout config
[
https://issues.apache.org/jira/browse/HIVE-27172?focusedWorklogId=859424&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859424
]
ASF GitHub Bot logged work on HIVE-27172:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 15:13
Start Date: 27/Apr/23 15:13
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4150:
URL: https://github.com/apache/hive/pull/4150#discussion_r1179316177
##
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStoreTimeout.java:
##
@@ -43,9 +46,10 @@ public class TestHiveMetaStoreTimeout {
protected static HiveMetaStoreClient client;
protected static Configuration conf;
protected static Warehouse warehouse;
+ protected static int port;
- @BeforeClass
- public static void setUp() throws Exception {
+ @Before
+ public void setUp() throws Exception {
HMSHandler.testTimeoutEnabled = true;
Review Comment:
do we need to repeat the whole setup for every test or it's possible to
extract just the HMSHandler part?
Issue Time Tracking
---
Worklog Id: (was: 859424)
Time Spent: 2h 20m (was: 2h 10m)
> Add the HMS client connection timeout config
>
>
> Key: HIVE-27172
> URL: https://issues.apache.org/jira/browse/HIVE-27172
> Project: Hive
> Issue Type: Task
> Components: Hive
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Currently {{HiveMetastoreClient}} use {{CLIENT_SOCKET_TIMEOUT}} as both
> socket timeout and connection timeout, it's not convenient for users to set a
> smaller connection timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27266) Retrieve only partNames if not need drop data in HMSHandler.dropPartitionsAndGetLocations
[
https://issues.apache.org/jira/browse/HIVE-27266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko resolved HIVE-27266.
---
Fix Version/s: 4.0.0
Resolution: Fixed
> Retrieve only partNames if not need drop data in
> HMSHandler.dropPartitionsAndGetLocations
> -
>
> Key: HIVE-27266
> URL: https://issues.apache.org/jira/browse/HIVE-27266
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Followed HIVE-19783, we only need partNames instead of partName and location
> pairs if we do not need check location.
> We add a new benchmark *dropTableMetadataWithPartitions* to delete only
> metadata rather than the real table data.
> Test results like:
> * Before the patch:
> {code:bash}
> Operation Mean Med Min Max Err%
> dropTableMetaOnlyWithPartitions.10 23.7021.8719.3631.7314.48
> dropTableMetaOnlyWithPartitions.100 54.4254.1545.9276.688.891
> dropTableMetaOnlyWithPartitions.1000 462.5456.1321.0654.315.96
> {code}
> * After the patch:
> {code:bash}
> Operation Mean Med Min Max Err%
> dropTableMetaOnlyWithPartitions.10 21.4921.2419.3027.906.661
> dropTableMetaOnlyWithPartitions.100 51.5148.3044.8685.2316.91
> dropTableMetaOnlyWithPartitions.1000 415.4407.2308.8595.214.28
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27266) Retrieve only partNames if not need drop data in HMSHandler.dropPartitionsAndGetLocations
[
https://issues.apache.org/jira/browse/HIVE-27266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717260#comment-17717260
]
Denys Kuzmenko commented on HIVE-27266:
---
Merged to master.
[~wechar] thanks for the patch!
> Retrieve only partNames if not need drop data in
> HMSHandler.dropPartitionsAndGetLocations
> -
>
> Key: HIVE-27266
> URL: https://issues.apache.org/jira/browse/HIVE-27266
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Followed HIVE-19783, we only need partNames instead of partName and location
> pairs if we do not need check location.
> We add a new benchmark *dropTableMetadataWithPartitions* to delete only
> metadata rather than the real table data.
> Test results like:
> * Before the patch:
> {code:bash}
> Operation Mean Med Min Max Err%
> dropTableMetaOnlyWithPartitions.10 23.7021.8719.3631.7314.48
> dropTableMetaOnlyWithPartitions.100 54.4254.1545.9276.688.891
> dropTableMetaOnlyWithPartitions.1000 462.5456.1321.0654.315.96
> {code}
> * After the patch:
> {code:bash}
> Operation Mean Med Min Max Err%
> dropTableMetaOnlyWithPartitions.10 21.4921.2419.3027.906.661
> dropTableMetaOnlyWithPartitions.100 51.5148.3044.8685.2316.91
> dropTableMetaOnlyWithPartitions.1000 415.4407.2308.8595.214.28
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27266) Retrieve only partNames if not need drop data in HMSHandler.dropPartitionsAndGetLocations
[
https://issues.apache.org/jira/browse/HIVE-27266?focusedWorklogId=859423&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859423
]
ASF GitHub Bot logged work on HIVE-27266:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 15:05
Start Date: 27/Apr/23 15:05
Worklog Time Spent: 10m
Work Description: deniskuzZ merged PR #4238:
URL: https://github.com/apache/hive/pull/4238
Issue Time Tracking
---
Worklog Id: (was: 859423)
Time Spent: 1h (was: 50m)
> Retrieve only partNames if not need drop data in
> HMSHandler.dropPartitionsAndGetLocations
> -
>
> Key: HIVE-27266
> URL: https://issues.apache.org/jira/browse/HIVE-27266
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Followed HIVE-19783, we only need partNames instead of partName and location
> pairs if we do not need check location.
> We add a new benchmark *dropTableMetadataWithPartitions* to delete only
> metadata rather than the real table data.
> Test results like:
> * Before the patch:
> {code:bash}
> Operation Mean Med Min Max Err%
> dropTableMetaOnlyWithPartitions.10 23.7021.8719.3631.7314.48
> dropTableMetaOnlyWithPartitions.100 54.4254.1545.9276.688.891
> dropTableMetaOnlyWithPartitions.1000 462.5456.1321.0654.315.96
> {code}
> * After the patch:
> {code:bash}
> Operation Mean Med Min Max Err%
> dropTableMetaOnlyWithPartitions.10 21.4921.2419.3027.906.661
> dropTableMetaOnlyWithPartitions.100 51.5148.3044.8685.2316.91
> dropTableMetaOnlyWithPartitions.1000 415.4407.2308.8595.214.28
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27287) Upgrade Commons-text to 1.10.0 to fix CVE
[
https://issues.apache.org/jira/browse/HIVE-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko resolved HIVE-27287.
---
Fix Version/s: 4.0.0
Resolution: Fixed
> Upgrade Commons-text to 1.10.0 to fix CVE
> -
>
> Key: HIVE-27287
> URL: https://issues.apache.org/jira/browse/HIVE-27287
> Project: Hive
> Issue Type: Improvement
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Apache Commons Text versions prior to 1.8 is vulnerable to
> [CVE-2022-42889|https://nvd.nist.gov/vuln/detail/CVE-2022-42889], which
> involves potential script execution when processing untrusted input using
> {{{}StringLookup{}}}. Direct and transitive references to Apache Commons Text
> prior to 1.10.0 should be upgraded to avoid the default interpolation
> behaviour.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27287) Upgrade Commons-text to 1.10.0 to fix CVE
[
https://issues.apache.org/jira/browse/HIVE-27287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717258#comment-17717258
]
Denys Kuzmenko commented on HIVE-27287:
---
Merged to master.
[~Aggarwal_Raghav] thanks for the contribution!
> Upgrade Commons-text to 1.10.0 to fix CVE
> -
>
> Key: HIVE-27287
> URL: https://issues.apache.org/jira/browse/HIVE-27287
> Project: Hive
> Issue Type: Improvement
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Apache Commons Text versions prior to 1.8 is vulnerable to
> [CVE-2022-42889|https://nvd.nist.gov/vuln/detail/CVE-2022-42889], which
> involves potential script execution when processing untrusted input using
> {{{}StringLookup{}}}. Direct and transitive references to Apache Commons Text
> prior to 1.10.0 should be upgraded to avoid the default interpolation
> behaviour.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27287) Upgrade Commons-text to 1.10.0 to fix CVE
[
https://issues.apache.org/jira/browse/HIVE-27287?focusedWorklogId=859422&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859422
]
ASF GitHub Bot logged work on HIVE-27287:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 15:00
Start Date: 27/Apr/23 15:00
Worklog Time Spent: 10m
Work Description: deniskuzZ merged PR #4260:
URL: https://github.com/apache/hive/pull/4260
Issue Time Tracking
---
Worklog Id: (was: 859422)
Time Spent: 1h 20m (was: 1h 10m)
> Upgrade Commons-text to 1.10.0 to fix CVE
> -
>
> Key: HIVE-27287
> URL: https://issues.apache.org/jira/browse/HIVE-27287
> Project: Hive
> Issue Type: Improvement
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Apache Commons Text versions prior to 1.8 is vulnerable to
> [CVE-2022-42889|https://nvd.nist.gov/vuln/detail/CVE-2022-42889], which
> involves potential script execution when processing untrusted input using
> {{{}StringLookup{}}}. Direct and transitive references to Apache Commons Text
> prior to 1.10.0 should be upgraded to avoid the default interpolation
> behaviour.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27292) Upgrade Zookeeper to 3.7.1
[ https://issues.apache.org/jira/browse/HIVE-27292?focusedWorklogId=859421&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859421 ] ASF GitHub Bot logged work on HIVE-27292: - Author: ASF GitHub Bot Created on: 27/Apr/23 14:56 Start Date: 27/Apr/23 14:56 Worklog Time Spent: 10m Work Description: deniskuzZ commented on PR #4264: URL: https://github.com/apache/hive/pull/4264#issuecomment-1525848855 @amanraj2520 should we wait for https://github.com/apache/hive/pull/4270? I've just re-triggered it Issue Time Tracking --- Worklog Id: (was: 859421) Time Spent: 1h 20m (was: 1h 10m) > Upgrade Zookeeper to 3.7.1 > -- > > Key: HIVE-27292 > URL: https://issues.apache.org/jira/browse/HIVE-27292 > Project: Hive > Issue Type: Bug >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Upgrade Zookeper from 3.6.3 to 3.7.1 since 3.6.3 is in end of life. > https://endoflife.date/zookeeper -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27303) select query result is different when enable/disable mapjoin with UNION ALL
Mahesh Raju Somalaraju created HIVE-27303: - Summary: select query result is different when enable/disable mapjoin with UNION ALL Key: HIVE-27303 URL: https://issues.apache.org/jira/browse/HIVE-27303 Project: Hive Issue Type: Bug Reporter: Mahesh Raju Somalaraju Assignee: Mahesh Raju Somalaraju select query result is different when enable/disable mapjoin with UNION ALL Below are the reproduce steps. As per query when map.join is disabled it should not give rows(duplicate). Same is working fine with map.join=true. Expected result: Empty rows. Problem: returning duplicate rows. Steps: -- SET hive.server2.tez.queue.access.check=true; SET tez.queue.name=default SET hive.query.results.cache.enabled=false; SET hive.fetch.task.conversion=none; SET hive.execution.engine=tez; SET hive.stats.autogather=true; SET hive.server2.enable.doAs=false; SET hive.auto.convert.join=true; drop table if exists hive1_tbl_data; drop table if exists hive2_tbl_data; drop table if exists hive3_tbl_data; drop table if exists hive4_tbl_data; CREATE EXTERNAL TABLE hive1_tbl_data (COLUMID string,COLUMN_FN string,COLUMN_LN string,EMAIL string,COL_UPDATED_DATE timestamp, PK_COLUM string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' TBLPROPERTIES ( 'TRANSLATED_TO_EXTERNAL'='true', 'bucketing_version'='2', 'external.table.purge'='true', 'parquet.compression'='SNAPPY'); CREATE EXTERNAL TABLE hive2_tbl_data (COLUMID string,COLUMN_FN string,COLUMN_LN string,EMAIL string,COL_UPDATED_DATE timestamp, PK_COLUM string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' TBLPROPERTIES ( 'TRANSLATED_TO_EXTERNAL'='true', 'bucketing_version'='2', 'external.table.purge'='true', 'parquet.compression'='SNAPPY'); CREATE EXTERNAL TABLE hive3_tbl_data (COLUMID string,COLUMN_FN string,COLUMN_LN string,EMAIL string,COL_UPDATED_DATE timestamp, PK_COLUM string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' TBLPROPERTIES ( 'TRANSLATED_TO_EXTERNAL'='true', 'bucketing_version'='2', 'external.table.purge'='true', 'parquet.compression'='SNAPPY'); CREATE EXTERNAL TABLE hive4_tbl_data (COLUMID string,COLUMN_FN string,COLUMN_LN string,EMAIL string,COL_UPDATED_DATE timestamp, PK_COLUM string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' TBLPROPERTIES ( 'TRANSLATED_TO_EXTERNAL'='true', 'bucketing_version'='2', 'external.table.purge'='true', 'parquet.compression'='SNAPPY'); insert into table hive1_tbl_data select '1','john','doe','j...@hotmail.com','2014-01-01 12:01:02','4000-1'; insert into table hive1_tbl_data select '2','john','doe','j...@hotmail.com','2014-01-01 12:01:02','4000-1';insert into table hive2_tbl_data select '1','john','doe','j...@hotmail.com','2014-01-01 12:01:02','1'; insert into table hive2_tbl_data select '2','john','doe','j...@hotmail.com','2014-01-01 12:01:02','1'; select t.COLUMID from ( select distinct t.COLUMID as COLUMID from (SELECT COLUMID FROM hive3_tbl_data UNION ALL SELECT COLUMID FROM hive1_tbl_data) t ) t left join ( select distinct t.COLUMID from (SELECT COLUMID FROM hive4_tbl_data UNI
[jira] [Work logged] (HIVE-27281) Add ability of masking to Beeline q-tests
[
https://issues.apache.org/jira/browse/HIVE-27281?focusedWorklogId=859419&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859419
]
ASF GitHub Bot logged work on HIVE-27281:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 14:51
Start Date: 27/Apr/23 14:51
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4254:
URL: https://github.com/apache/hive/pull/4254#discussion_r1179282848
##
itests/util/src/main/java/org/apache/hadoop/hive/cli/control/CoreBeeLineDriver.java:
##
@@ -97,6 +101,8 @@ public CoreBeeLineDriver(AbstractCliConfig testCliConfig) {
initScript = new File(testScriptDirectory,
testCliConfig.getInitScript());
}
cleanupScript = new File(testScriptDirectory,
testCliConfig.getCleanupScript());
+this.replaceHandler = new QTestReplaceHandler();
Review Comment:
Could we use QTestUtil as in other drivers?
Issue Time Tracking
---
Worklog Id: (was: 859419)
Time Spent: 1h 10m (was: 1h)
> Add ability of masking to Beeline q-tests
> -
>
> Key: HIVE-27281
> URL: https://issues.apache.org/jira/browse/HIVE-27281
> Project: Hive
> Issue Type: Improvement
>Reporter: Dmitriy Fingerman
>Assignee: Dmitriy Fingerman
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27273) Iceberg: Upgrade iceberg to 1.2.1
[
https://issues.apache.org/jira/browse/HIVE-27273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-27273:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Iceberg: Upgrade iceberg to 1.2.1
> --
>
> Key: HIVE-27273
> URL: https://issues.apache.org/jira/browse/HIVE-27273
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> [https://iceberg.apache.org/releases/#121-release] Iceberg1.2.1(include
> 1.2.0) has lots of improvement, e.g. _branch commit_ and
> _{{position_deletes}} metadata table._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27273) Iceberg: Upgrade iceberg to 1.2.1
[
https://issues.apache.org/jira/browse/HIVE-27273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-27273:
--
Fix Version/s: 4.0.0
Status: Patch Available (was: Open)
> Iceberg: Upgrade iceberg to 1.2.1
> --
>
> Key: HIVE-27273
> URL: https://issues.apache.org/jira/browse/HIVE-27273
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> [https://iceberg.apache.org/releases/#121-release] Iceberg1.2.1(include
> 1.2.0) has lots of improvement, e.g. _branch commit_ and
> _{{position_deletes}} metadata table._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27273) Iceberg: Upgrade iceberg to 1.2.1
[
https://issues.apache.org/jira/browse/HIVE-27273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717250#comment-17717250
]
Denys Kuzmenko commented on HIVE-27273:
---
Merged to master.
[~zhangbutao] thanks for the patch and [~InvisibleProgrammer], [~pvary] for the
review!
> Iceberg: Upgrade iceberg to 1.2.1
> --
>
> Key: HIVE-27273
> URL: https://issues.apache.org/jira/browse/HIVE-27273
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> [https://iceberg.apache.org/releases/#121-release] Iceberg1.2.1(include
> 1.2.0) has lots of improvement, e.g. _branch commit_ and
> _{{position_deletes}} metadata table._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27273) Iceberg: Upgrade iceberg to 1.2.1
[
https://issues.apache.org/jira/browse/HIVE-27273?focusedWorklogId=859415&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859415
]
ASF GitHub Bot logged work on HIVE-27273:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 14:32
Start Date: 27/Apr/23 14:32
Worklog Time Spent: 10m
Work Description: deniskuzZ merged PR #4252:
URL: https://github.com/apache/hive/pull/4252
Issue Time Tracking
---
Worklog Id: (was: 859415)
Time Spent: 2h 40m (was: 2.5h)
> Iceberg: Upgrade iceberg to 1.2.1
> --
>
> Key: HIVE-27273
> URL: https://issues.apache.org/jira/browse/HIVE-27273
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> [https://iceberg.apache.org/releases/#121-release] Iceberg1.2.1(include
> 1.2.0) has lots of improvement, e.g. _branch commit_ and
> _{{position_deletes}} metadata table._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27273) Iceberg: Upgrade iceberg to 1.2.1
[
https://issues.apache.org/jira/browse/HIVE-27273?focusedWorklogId=859414&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859414
]
ASF GitHub Bot logged work on HIVE-27273:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 14:30
Start Date: 27/Apr/23 14:30
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on PR #4252:
URL: https://github.com/apache/hive/pull/4252#issuecomment-1525802452
@zhangbutao, @InvisibleProgrammer could we please create a backport ticket
that includes:
1. 37cb7caa75e7419a683f84793b77190f4cd3c899 - MR: Skip filter translation if
there are no leaves;
2. c3232b664745ebf761b6a74f4c5b55cc48bfd209: Hive: Use EnvironmentContext
instead of Hive Locks to provide transactional commits after HIVE-26882;
3. 333227fbd13821365cec1bdbfcb9314a239bea0f - Hive: Refactor commit lock
mechanism from HiveTableOperations;
4. fede493d59f17ff2bfc0744b296d90bd36130386 - Hive: Lock hardening
+ what is in Zsolt's list
Issue Time Tracking
---
Worklog Id: (was: 859414)
Time Spent: 2.5h (was: 2h 20m)
> Iceberg: Upgrade iceberg to 1.2.1
> --
>
> Key: HIVE-27273
> URL: https://issues.apache.org/jira/browse/HIVE-27273
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> [https://iceberg.apache.org/releases/#121-release] Iceberg1.2.1(include
> 1.2.0) has lots of improvement, e.g. _branch commit_ and
> _{{position_deletes}} metadata table._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27290) Upgrade com.jayway.jsonpath » json-path to 2.8.0 to fix CVEs
[ https://issues.apache.org/jira/browse/HIVE-27290?focusedWorklogId=859413&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859413 ] ASF GitHub Bot logged work on HIVE-27290: - Author: ASF GitHub Bot Created on: 27/Apr/23 14:13 Start Date: 27/Apr/23 14:13 Worklog Time Spent: 10m Work Description: veghlaci05 merged PR #4266: URL: https://github.com/apache/hive/pull/4266 Issue Time Tracking --- Worklog Id: (was: 859413) Time Spent: 50m (was: 40m) > Upgrade com.jayway.jsonpath » json-path to 2.8.0 to fix CVEs > > > Key: HIVE-27290 > URL: https://issues.apache.org/jira/browse/HIVE-27290 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27197) Iceberg: Support Iceberg version travel by reference name
[ https://issues.apache.org/jira/browse/HIVE-27197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko resolved HIVE-27197. --- Fix Version/s: 4.0.0 Resolution: Fixed > Iceberg: Support Iceberg version travel by reference name > --- > > Key: HIVE-27197 > URL: https://issues.apache.org/jira/browse/HIVE-27197 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: zhangbutao >Assignee: zhangbutao >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > This ticket is inspired by https://github.com/apache/iceberg/pull/6575 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27197) Iceberg: Support Iceberg version travel by reference name
[ https://issues.apache.org/jira/browse/HIVE-27197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717238#comment-17717238 ] Denys Kuzmenko commented on HIVE-27197: --- Merged to master. [~zhangbutao] thanks for the contribution! > Iceberg: Support Iceberg version travel by reference name > --- > > Key: HIVE-27197 > URL: https://issues.apache.org/jira/browse/HIVE-27197 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: zhangbutao >Assignee: zhangbutao >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > This ticket is inspired by https://github.com/apache/iceberg/pull/6575 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27197) Iceberg: Support Iceberg version travel by reference name
[ https://issues.apache.org/jira/browse/HIVE-27197?focusedWorklogId=859411&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859411 ] ASF GitHub Bot logged work on HIVE-27197: - Author: ASF GitHub Bot Created on: 27/Apr/23 14:05 Start Date: 27/Apr/23 14:05 Worklog Time Spent: 10m Work Description: deniskuzZ merged PR #4173: URL: https://github.com/apache/hive/pull/4173 Issue Time Tracking --- Worklog Id: (was: 859411) Time Spent: 1h 10m (was: 1h) > Iceberg: Support Iceberg version travel by reference name > --- > > Key: HIVE-27197 > URL: https://issues.apache.org/jira/browse/HIVE-27197 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: zhangbutao >Assignee: zhangbutao >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > This ticket is inspired by https://github.com/apache/iceberg/pull/6575 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27302) Iceberg: Suport write to iceberg branch
[ https://issues.apache.org/jira/browse/HIVE-27302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangbutao reassigned HIVE-27302: - Assignee: zhangbutao > Iceberg: Suport write to iceberg branch > --- > > Key: HIVE-27302 > URL: https://issues.apache.org/jira/browse/HIVE-27302 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: zhangbutao >Assignee: zhangbutao >Priority: Major > > This feature depends on Iceberg1.2.0 interface: > [https://github.com/apache/iceberg/pull/5234] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27302) Iceberg: Suport write to iceberg branch
zhangbutao created HIVE-27302: - Summary: Iceberg: Suport write to iceberg branch Key: HIVE-27302 URL: https://issues.apache.org/jira/browse/HIVE-27302 Project: Hive Issue Type: Sub-task Components: Iceberg integration Reporter: zhangbutao This feature depends on Iceberg1.2.0 interface: [https://github.com/apache/iceberg/pull/5234] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27277) Set up github actions workflow to build and push docker image to docker hub
[
https://issues.apache.org/jira/browse/HIVE-27277?focusedWorklogId=859407&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859407
]
ASF GitHub Bot logged work on HIVE-27277:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 13:12
Start Date: 27/Apr/23 13:12
Worklog Time Spent: 10m
Work Description: simhadri-g commented on code in PR #4274:
URL: https://github.com/apache/hive/pull/4274#discussion_r1179136508
##
.github/workflows/docker-image.yml:
##
@@ -0,0 +1,38 @@
+name: ci hive docker image
+
+on:
+ push:
+branches:
+ - "master"
+
+jobs:
+ build:
+runs-on: ubuntu-latest
+steps:
+ -
+name: Checkout
+uses: actions/checkout@v3
+ -
+name: Login to Docker Hub
+uses: docker/login-action@v2
+with:
+ username: ${{ secrets.DOCKERHUB_USERNAME }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ -
+name: Set up Docker Buildx
+uses: docker/setup-buildx-action@v2
+
+ -
+name: Build and push
+uses: docker/build-push-action@v4
+with:
+ context: ./packaging/src/docker/
+ file: ./packaging/src/docker/Dockerfile
+ push: true
+ tags: ${{ secrets.DOCKERHUB_USERNAME }}/hive:test-image
Review Comment:
For GA: the versions that are set in the .yml file were manually configured
after looking at the hive/pom.xml file.
For hive:daily, i think we can obtain them from the pom.xml file
Issue Time Tracking
---
Worklog Id: (was: 859407)
Time Spent: 1h 40m (was: 1.5h)
> Set up github actions workflow to build and push docker image to docker hub
> ---
>
> Key: HIVE-27277
> URL: https://issues.apache.org/jira/browse/HIVE-27277
> Project: Hive
> Issue Type: Sub-task
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27198) Delete directly aborted transactions instead of select and loading ids
[
https://issues.apache.org/jira/browse/HIVE-27198?focusedWorklogId=859408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859408
]
ASF GitHub Bot logged work on HIVE-27198:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 13:12
Start Date: 27/Apr/23 13:12
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4174:
URL: https://github.com/apache/hive/pull/4174#discussion_r1179137448
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##
@@ -888,35 +887,13 @@ public void cleanEmptyAbortedAndCommittedTxns() throws
MetaException {
*/
long lowWaterMark = getOpenTxnTimeoutLowBoundaryTxnId(dbConn);
-String s = "SELECT \"TXN_ID\" FROM \"TXNS\" WHERE " +
+String s = "DELETE FROM \"TXNS\" WHERE " +
Review Comment:
Could we move sql to constants (see `TxnQueries`) and use prepared statement?
Issue Time Tracking
---
Worklog Id: (was: 859408)
Time Spent: 1h 20m (was: 1h 10m)
> Delete directly aborted transactions instead of select and loading ids
> --
>
> Key: HIVE-27198
> URL: https://issues.apache.org/jira/browse/HIVE-27198
> Project: Hive
> Issue Type: Improvement
>Reporter: Mahesh Raju Somalaraju
>Assignee: Mahesh Raju Somalaraju
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> in cleaning the aborted transaction , we can directly deletes the txns
> instead of selecting and process.
> method name:
> cleanEmptyAbortedAndCommittedTxns
> Code:
> String s = "SELECT \"TXN_ID\" FROM \"TXNS\" WHERE " +
> "\"TXN_ID\" NOT IN (SELECT \"TC_TXNID\" FROM \"TXN_COMPONENTS\") AND " +
> " (\"TXN_STATE\" = " + TxnStatus.ABORTED + " OR \"TXN_STATE\" = " +
> TxnStatus.COMMITTED + ") AND "
> + " \"TXN_ID\" < " + lowWaterMark;
>
> proposed code:
> String s = "DELETE \"TXN_ID\" FROM \"TXNS\" WHERE " +
> "\"TXN_ID\" NOT IN (SELECT \"TC_TXNID\" FROM \"TXN_COMPONENTS\") AND " +
> " (\"TXN_STATE\" = " + TxnStatus.ABORTED + " OR \"TXN_STATE\" = " +
> TxnStatus.COMMITTED + ") AND "
> + " \"TXN_ID\" < " + lowWaterMark;
>
> the select needs to be eliminated and the delete should work with the where
> clause instead of the built in clause
> we can see no reason for loading the ids into memory and then generate a huge
> sql
>
> Bathcing is also not necessary here, we can deletes the records directly
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27277) Set up github actions workflow to build and push docker image to docker hub
[ https://issues.apache.org/jira/browse/HIVE-27277?focusedWorklogId=859406&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859406 ] ASF GitHub Bot logged work on HIVE-27277: - Author: ASF GitHub Bot Created on: 27/Apr/23 13:11 Start Date: 27/Apr/23 13:11 Worklog Time Spent: 10m Work Description: simhadri-g commented on code in PR #4274: URL: https://github.com/apache/hive/pull/4274#discussion_r1179135961 ## .github/workflows/docker-image.yml: ## @@ -0,0 +1,38 @@ +name: ci hive docker image + +on: + push: Review Comment: I think we should have 2 workflows: 1. GA workflow - Frequency : Once per release 2. For the latest dev images. - Frequency: Once per week? Because on an average hive gets about 10 to 15 commits per week.(https://github.com/apache/hive/graphs/commit-activity) This PR set up a workflow to build and publish docker images for the GA versions of hive. I will raise a follow-up jira to address the workflow needed for daily/dev images. Issue Time Tracking --- Worklog Id: (was: 859406) Time Spent: 1.5h (was: 1h 20m) > Set up github actions workflow to build and push docker image to docker hub > --- > > Key: HIVE-27277 > URL: https://issues.apache.org/jira/browse/HIVE-27277 > Project: Hive > Issue Type: Sub-task >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27198) Delete directly aborted transactions instead of select and loading ids
[
https://issues.apache.org/jira/browse/HIVE-27198?focusedWorklogId=859405&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859405
]
ASF GitHub Bot logged work on HIVE-27198:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 13:10
Start Date: 27/Apr/23 13:10
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #4174:
URL: https://github.com/apache/hive/pull/4174#discussion_r1179133565
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java:
##
@@ -875,7 +875,6 @@ public void cleanEmptyAbortedAndCommittedTxns() throws
MetaException {
try {
Connection dbConn = null;
Statement stmt = null;
- ResultSet rs = null;
Review Comment:
please use `try-with-resources` instead of explicit resource management
Issue Time Tracking
---
Worklog Id: (was: 859405)
Time Spent: 1h 10m (was: 1h)
> Delete directly aborted transactions instead of select and loading ids
> --
>
> Key: HIVE-27198
> URL: https://issues.apache.org/jira/browse/HIVE-27198
> Project: Hive
> Issue Type: Improvement
>Reporter: Mahesh Raju Somalaraju
>Assignee: Mahesh Raju Somalaraju
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> in cleaning the aborted transaction , we can directly deletes the txns
> instead of selecting and process.
> method name:
> cleanEmptyAbortedAndCommittedTxns
> Code:
> String s = "SELECT \"TXN_ID\" FROM \"TXNS\" WHERE " +
> "\"TXN_ID\" NOT IN (SELECT \"TC_TXNID\" FROM \"TXN_COMPONENTS\") AND " +
> " (\"TXN_STATE\" = " + TxnStatus.ABORTED + " OR \"TXN_STATE\" = " +
> TxnStatus.COMMITTED + ") AND "
> + " \"TXN_ID\" < " + lowWaterMark;
>
> proposed code:
> String s = "DELETE \"TXN_ID\" FROM \"TXNS\" WHERE " +
> "\"TXN_ID\" NOT IN (SELECT \"TC_TXNID\" FROM \"TXN_COMPONENTS\") AND " +
> " (\"TXN_STATE\" = " + TxnStatus.ABORTED + " OR \"TXN_STATE\" = " +
> TxnStatus.COMMITTED + ") AND "
> + " \"TXN_ID\" < " + lowWaterMark;
>
> the select needs to be eliminated and the delete should work with the where
> clause instead of the built in clause
> we can see no reason for loading the ids into memory and then generate a huge
> sql
>
> Bathcing is also not necessary here, we can deletes the records directly
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27277) Set up github actions workflow to build and push docker image to docker hub
[ https://issues.apache.org/jira/browse/HIVE-27277?focusedWorklogId=859403&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859403 ] ASF GitHub Bot logged work on HIVE-27277: - Author: ASF GitHub Bot Created on: 27/Apr/23 12:57 Start Date: 27/Apr/23 12:57 Worklog Time Spent: 10m Work Description: simhadri-g commented on code in PR #4274: URL: https://github.com/apache/hive/pull/4274#discussion_r1179117576 ## packaging/src/docker/Dockerfile: ## @@ -14,14 +14,31 @@ # See the License for the specific language governing permissions and # limitations under the License. # +ARG BUILD_ENV + FROM ubuntu as unarchive Review Comment: Done Issue Time Tracking --- Worklog Id: (was: 859403) Time Spent: 1h 20m (was: 1h 10m) > Set up github actions workflow to build and push docker image to docker hub > --- > > Key: HIVE-27277 > URL: https://issues.apache.org/jira/browse/HIVE-27277 > Project: Hive > Issue Type: Sub-task >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27299) Upgrade guava version to 31.1-jre to fix CVE
[ https://issues.apache.org/jira/browse/HIVE-27299?focusedWorklogId=859399&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859399 ] ASF GitHub Bot logged work on HIVE-27299: - Author: ASF GitHub Bot Created on: 27/Apr/23 11:57 Start Date: 27/Apr/23 11:57 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4271: URL: https://github.com/apache/hive/pull/4271#issuecomment-1525562356 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4271) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4271&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4271&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4271&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4271&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4271&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4271&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4271&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4271&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4271&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4271&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4271&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4271&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4271&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4271&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 859399) Time Spent: 50m (was: 40m) > Upgrade guava version to 31.1-jre to fix CVE > > > Key: HIVE-27299 > URL: https://issues.apache.org/jira/browse/HIVE-27299 > Project: Hive > Issue Type: Improvement >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The guava version in Hive master branch is 22.0 which has 2 Direct CVE: > # > [CVE-2020-8908|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908] > # > [CVE-2018-10237|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10237] > Component like Tez 0.10.2 (used in hive) has also moved to 31.1-jre. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26779) UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent
[
https://issues.apache.org/jira/browse/HIVE-26779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa resolved HIVE-26779.
---
Resolution: Fixed
Merged to master. Thanks [~dkuzmenko] for review.
> UNION ALL throws SemanticException when trying to remove partition
> predicates: fail to find child from parent
> -
>
> Key: HIVE-26779
> URL: https://issues.apache.org/jira/browse/HIVE-26779
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 4.0.0-alpha-2
>Reporter: Zhizhen Hou
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {quote}Reproduce sql:
> drop table if exists tez_test_t1;
> create table tez_test_t1(md_exper string);
> insert into tez_test_t1 values('tez_test_t1-md_expr');
> drop table if exists tez_test_t5;
> create table tez_test_t5(md_exper string, did string);
> insert into tez_test_t5 values('tez_test_t5-md_expr','tez_test_t5-did');
> drop table if exists tez_test_t2;
> create table tez_test_t2(did string);
> insert into tez_test_t2 values('tez_test_t2-did');
> SELECT md_exper,null as ads_h5_gap , null as first_login_did, null as
> inclick_did
> FROM tez_test_t1
> UNION ALL
> SELECT md_exper, ads_h5_gap ,
> NULL AS first_login_did,did AS inclick_did
> FROM tez_test_t5
> LATERAL VIEW explode(split('0,6', ',')) gaps AS ads_h5_gap
> UNION ALL
> SELECT '' AS md_exper,'0,6' as ads_h5_gap ,
> did AS first_login_did, NULL AS inclick_did
> FROM tez_test_t2
> GROUP BY did;
> {quote}
>
> StackTrace
> 2022-11-27T09:31:06,801 ERROR [21d35a7f-9625-46ae-9c3d-13ca925f55cb main]:
> ql.Driver (:()) - FAILED: SemanticException Exception when trying to remove
> partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to
> remove partition predicates: fail to find child from parent
> at
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:859)
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:348)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:573)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:241)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12333)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:286)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26779) UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent
[
https://issues.apache.org/jira/browse/HIVE-26779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa reassigned HIVE-26779:
-
Assignee: Krisztian Kasa (was: Zhizhen Hou)
> UNION ALL throws SemanticException when trying to remove partition
> predicates: fail to find child from parent
> -
>
> Key: HIVE-26779
> URL: https://issues.apache.org/jira/browse/HIVE-26779
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 4.0.0-alpha-2
>Reporter: Zhizhen Hou
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {quote}Reproduce sql:
> drop table if exists tez_test_t1;
> create table tez_test_t1(md_exper string);
> insert into tez_test_t1 values('tez_test_t1-md_expr');
> drop table if exists tez_test_t5;
> create table tez_test_t5(md_exper string, did string);
> insert into tez_test_t5 values('tez_test_t5-md_expr','tez_test_t5-did');
> drop table if exists tez_test_t2;
> create table tez_test_t2(did string);
> insert into tez_test_t2 values('tez_test_t2-did');
> SELECT md_exper,null as ads_h5_gap , null as first_login_did, null as
> inclick_did
> FROM tez_test_t1
> UNION ALL
> SELECT md_exper, ads_h5_gap ,
> NULL AS first_login_did,did AS inclick_did
> FROM tez_test_t5
> LATERAL VIEW explode(split('0,6', ',')) gaps AS ads_h5_gap
> UNION ALL
> SELECT '' AS md_exper,'0,6' as ads_h5_gap ,
> did AS first_login_did, NULL AS inclick_did
> FROM tez_test_t2
> GROUP BY did;
> {quote}
>
> StackTrace
> 2022-11-27T09:31:06,801 ERROR [21d35a7f-9625-46ae-9c3d-13ca925f55cb main]:
> ql.Driver (:()) - FAILED: SemanticException Exception when trying to remove
> partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to
> remove partition predicates: fail to find child from parent
> at
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:859)
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:348)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:573)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:241)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12333)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:286)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26779) UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent
[
https://issues.apache.org/jira/browse/HIVE-26779?focusedWorklogId=859397&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859397
]
ASF GitHub Bot logged work on HIVE-26779:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 11:49
Start Date: 27/Apr/23 11:49
Worklog Time Spent: 10m
Work Description: kasakrisz merged PR #4272:
URL: https://github.com/apache/hive/pull/4272
Issue Time Tracking
---
Worklog Id: (was: 859397)
Time Spent: 50m (was: 40m)
> UNION ALL throws SemanticException when trying to remove partition
> predicates: fail to find child from parent
> -
>
> Key: HIVE-26779
> URL: https://issues.apache.org/jira/browse/HIVE-26779
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 4.0.0-alpha-2
>Reporter: Zhizhen Hou
>Assignee: Zhizhen Hou
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {quote}Reproduce sql:
> drop table if exists tez_test_t1;
> create table tez_test_t1(md_exper string);
> insert into tez_test_t1 values('tez_test_t1-md_expr');
> drop table if exists tez_test_t5;
> create table tez_test_t5(md_exper string, did string);
> insert into tez_test_t5 values('tez_test_t5-md_expr','tez_test_t5-did');
> drop table if exists tez_test_t2;
> create table tez_test_t2(did string);
> insert into tez_test_t2 values('tez_test_t2-did');
> SELECT md_exper,null as ads_h5_gap , null as first_login_did, null as
> inclick_did
> FROM tez_test_t1
> UNION ALL
> SELECT md_exper, ads_h5_gap ,
> NULL AS first_login_did,did AS inclick_did
> FROM tez_test_t5
> LATERAL VIEW explode(split('0,6', ',')) gaps AS ads_h5_gap
> UNION ALL
> SELECT '' AS md_exper,'0,6' as ads_h5_gap ,
> did AS first_login_did, NULL AS inclick_did
> FROM tez_test_t2
> GROUP BY did;
> {quote}
>
> StackTrace
> 2022-11-27T09:31:06,801 ERROR [21d35a7f-9625-46ae-9c3d-13ca925f55cb main]:
> ql.Driver (:()) - FAILED: SemanticException Exception when trying to remove
> partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to
> remove partition predicates: fail to find child from parent
> at
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:859)
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:348)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:573)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:241)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12333)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:286)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26779) UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent
[
https://issues.apache.org/jira/browse/HIVE-26779?focusedWorklogId=859396&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859396
]
ASF GitHub Bot logged work on HIVE-26779:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 11:46
Start Date: 27/Apr/23 11:46
Worklog Time Spent: 10m
Work Description: kasakrisz commented on PR #4272:
URL: https://github.com/apache/hive/pull/4272#issuecomment-1525549613
@aturoczy
I agree. But this one is just 972 lines :) Look at `SemanticAnalyzer`.
I'm not aware of any ongoing or planned refactor.
Since one of the inputs of these methods is a query plan writing unit tests
requires some plan generating logic which gives us the ability to define plans
in an easily understand way.
For example
* calling the parser and compiler on a test sql
* or implement utility methods to build a Hive operator tree similar to
Calcite `RelBuilder`.
Issue Time Tracking
---
Worklog Id: (was: 859396)
Time Spent: 40m (was: 0.5h)
> UNION ALL throws SemanticException when trying to remove partition
> predicates: fail to find child from parent
> -
>
> Key: HIVE-26779
> URL: https://issues.apache.org/jira/browse/HIVE-26779
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 4.0.0-alpha-2
>Reporter: Zhizhen Hou
>Assignee: Zhizhen Hou
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {quote}Reproduce sql:
> drop table if exists tez_test_t1;
> create table tez_test_t1(md_exper string);
> insert into tez_test_t1 values('tez_test_t1-md_expr');
> drop table if exists tez_test_t5;
> create table tez_test_t5(md_exper string, did string);
> insert into tez_test_t5 values('tez_test_t5-md_expr','tez_test_t5-did');
> drop table if exists tez_test_t2;
> create table tez_test_t2(did string);
> insert into tez_test_t2 values('tez_test_t2-did');
> SELECT md_exper,null as ads_h5_gap , null as first_login_did, null as
> inclick_did
> FROM tez_test_t1
> UNION ALL
> SELECT md_exper, ads_h5_gap ,
> NULL AS first_login_did,did AS inclick_did
> FROM tez_test_t5
> LATERAL VIEW explode(split('0,6', ',')) gaps AS ads_h5_gap
> UNION ALL
> SELECT '' AS md_exper,'0,6' as ads_h5_gap ,
> did AS first_login_did, NULL AS inclick_did
> FROM tez_test_t2
> GROUP BY did;
> {quote}
>
> StackTrace
> 2022-11-27T09:31:06,801 ERROR [21d35a7f-9625-46ae-9c3d-13ca925f55cb main]:
> ql.Driver (:()) - FAILED: SemanticException Exception when trying to remove
> partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to
> remove partition predicates: fail to find child from parent
> at
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:859)
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:348)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:573)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:241)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12333)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:286)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27294) Remove redundant qt_database_all.q for memory consumption reasons
[
https://issues.apache.org/jira/browse/HIVE-27294?focusedWorklogId=859389&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859389
]
ASF GitHub Bot logged work on HIVE-27294:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 10:59
Start Date: 27/Apr/23 10:59
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4267:
URL: https://github.com/apache/hive/pull/4267#issuecomment-1525480712
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4267)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4267&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4267&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4267&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=CODE_SMELL)
[1 Code
Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4267&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4267&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4267&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 859389)
Time Spent: 1h (was: 50m)
> Remove redundant qt_database_all.q for memory consumption reasons
> -
>
> Key: HIVE-27294
> URL: https://issues.apache.org/jira/browse/HIVE-27294
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently, while running qt_database_all.q the qtest environment starts and
> runs all the RDMBS docker containers at the same time in beforeTest, which
> might end up in extreme memory consumption. This is suboptimal, and
> considering that the test cases are all covered by single, separate qtests,
> we can simply remove qt_database_all.q.
> {code}
> ./ql/src/test/queries/clientpositive/qt_database_postgres.q
> ./ql/src/test/queries/clientpositive/qt_database_oracle.q
> ./ql/src/test/queries/clientpositive/qt_database_mssql.q
> ./ql/src/test/queries/clientpositive/qt_database_mariad
[jira] [Work logged] (HIVE-27299) Upgrade guava version to 31.1-jre to fix CVE
[ https://issues.apache.org/jira/browse/HIVE-27299?focusedWorklogId=859382&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859382 ] ASF GitHub Bot logged work on HIVE-27299: - Author: ASF GitHub Bot Created on: 27/Apr/23 10:42 Start Date: 27/Apr/23 10:42 Worklog Time Spent: 10m Work Description: Aggarwal-Raghav commented on PR #4271: URL: https://github.com/apache/hive/pull/4271#issuecomment-1525445127 Thanks for letting me know @ayushtkn. I was not aware of this. Issue Time Tracking --- Worklog Id: (was: 859382) Time Spent: 40m (was: 0.5h) > Upgrade guava version to 31.1-jre to fix CVE > > > Key: HIVE-27299 > URL: https://issues.apache.org/jira/browse/HIVE-27299 > Project: Hive > Issue Type: Improvement >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The guava version in Hive master branch is 22.0 which has 2 Direct CVE: > # > [CVE-2020-8908|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908] > # > [CVE-2018-10237|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10237] > Component like Tez 0.10.2 (used in hive) has also moved to 31.1-jre. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27299) Upgrade guava version to 31.1-jre to fix CVE
[ https://issues.apache.org/jira/browse/HIVE-27299?focusedWorklogId=859379&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859379 ] ASF GitHub Bot logged work on HIVE-27299: - Author: ASF GitHub Bot Created on: 27/Apr/23 10:32 Start Date: 27/Apr/23 10:32 Worklog Time Spent: 10m Work Description: ayushtkn commented on PR #4271: URL: https://github.com/apache/hive/pull/4271#issuecomment-1525418602 have played with guava in the past, to chase hadoop upgrade. but hive has too many dependencies with different guava versions, so that didn't work well. So, ultimately I shaded guava in hadoop for hive and we moved to the hadoop version in which guava was shaded. @Aggarwal-Raghav no point running the tests again and again those are genuine errors, guava upgrade ain't just a change in the pom Issue Time Tracking --- Worklog Id: (was: 859379) Time Spent: 0.5h (was: 20m) > Upgrade guava version to 31.1-jre to fix CVE > > > Key: HIVE-27299 > URL: https://issues.apache.org/jira/browse/HIVE-27299 > Project: Hive > Issue Type: Improvement >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > The guava version in Hive master branch is 22.0 which has 2 Direct CVE: > # > [CVE-2020-8908|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908] > # > [CVE-2018-10237|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10237] > Component like Tez 0.10.2 (used in hive) has also moved to 31.1-jre. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27299) Upgrade guava version to 31.1-jre to fix CVE
[ https://issues.apache.org/jira/browse/HIVE-27299?focusedWorklogId=859377&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859377 ] ASF GitHub Bot logged work on HIVE-27299: - Author: ASF GitHub Bot Created on: 27/Apr/23 10:23 Start Date: 27/Apr/23 10:23 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #4271: URL: https://github.com/apache/hive/pull/4271#issuecomment-1525398100 It is a fair change to fix the CVE's, but I have doubt it will be as easy as the change. The Guava 16 was released in 2014, the 31 is released somewhere in early 2023. After 9 years, I think there will be several breaking change that needs to adjust in the hive side. It should be a huge, but respectful task. cc: @abstractdog @ayushtkn @deniskuzZ Issue Time Tracking --- Worklog Id: (was: 859377) Time Spent: 20m (was: 10m) > Upgrade guava version to 31.1-jre to fix CVE > > > Key: HIVE-27299 > URL: https://issues.apache.org/jira/browse/HIVE-27299 > Project: Hive > Issue Type: Improvement >Reporter: Raghav Aggarwal >Assignee: Raghav Aggarwal >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The guava version in Hive master branch is 22.0 which has 2 Direct CVE: > # > [CVE-2020-8908|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908] > # > [CVE-2018-10237|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10237] > Component like Tez 0.10.2 (used in hive) has also moved to 31.1-jre. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26779) UNION ALL throws SemanticException when trying to remove partition predicates: fail to find child from parent
[
https://issues.apache.org/jira/browse/HIVE-26779?focusedWorklogId=859374&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859374
]
ASF GitHub Bot logged work on HIVE-26779:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 10:14
Start Date: 27/Apr/23 10:14
Worklog Time Spent: 10m
Work Description: TuroczyX commented on PR #4272:
URL: https://github.com/apache/hive/pull/4272#issuecomment-1525375405
I'm ok with this change and make sense. But the GenTezUtils is insanely big
a complex. The cyclomatic complexity is too high to have a proper unit test for
this class. Is there any intention to refactor this class, because this class
is hurting closely all SOLID principals.
Otherwise the change and the tests are OK to me.
Issue Time Tracking
---
Worklog Id: (was: 859374)
Time Spent: 0.5h (was: 20m)
> UNION ALL throws SemanticException when trying to remove partition
> predicates: fail to find child from parent
> -
>
> Key: HIVE-26779
> URL: https://issues.apache.org/jira/browse/HIVE-26779
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 4.0.0-alpha-2
>Reporter: Zhizhen Hou
>Assignee: Zhizhen Hou
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {quote}Reproduce sql:
> drop table if exists tez_test_t1;
> create table tez_test_t1(md_exper string);
> insert into tez_test_t1 values('tez_test_t1-md_expr');
> drop table if exists tez_test_t5;
> create table tez_test_t5(md_exper string, did string);
> insert into tez_test_t5 values('tez_test_t5-md_expr','tez_test_t5-did');
> drop table if exists tez_test_t2;
> create table tez_test_t2(did string);
> insert into tez_test_t2 values('tez_test_t2-did');
> SELECT md_exper,null as ads_h5_gap , null as first_login_did, null as
> inclick_did
> FROM tez_test_t1
> UNION ALL
> SELECT md_exper, ads_h5_gap ,
> NULL AS first_login_did,did AS inclick_did
> FROM tez_test_t5
> LATERAL VIEW explode(split('0,6', ',')) gaps AS ads_h5_gap
> UNION ALL
> SELECT '' AS md_exper,'0,6' as ads_h5_gap ,
> did AS first_login_did, NULL AS inclick_did
> FROM tez_test_t2
> GROUP BY did;
> {quote}
>
> StackTrace
> 2022-11-27T09:31:06,801 ERROR [21d35a7f-9625-46ae-9c3d-13ca925f55cb main]:
> ql.Driver (:()) - FAILED: SemanticException Exception when trying to remove
> partition predicates: fail to find child from parent
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception when trying to
> remove partition predicates: fail to find child from parent
> at
> org.apache.hadoop.hive.ql.exec.Operator.removeChildAndAdoptItsChildren(Operator.java:859)
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.removeUnionOperators(GenTezUtils.java:348)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:573)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:241)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12333)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:286)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27290) Upgrade com.jayway.jsonpath » json-path to 2.8.0 to fix CVEs
[ https://issues.apache.org/jira/browse/HIVE-27290?focusedWorklogId=859370&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859370 ] ASF GitHub Bot logged work on HIVE-27290: - Author: ASF GitHub Bot Created on: 27/Apr/23 09:59 Start Date: 27/Apr/23 09:59 Worklog Time Spent: 10m Work Description: TuroczyX commented on PR #4266: URL: https://github.com/apache/hive/pull/4266#issuecomment-1525332526 @deniskuzZ @veghlaci05 is it possible to merge? Spell check is skipped. Issue Time Tracking --- Worklog Id: (was: 859370) Time Spent: 40m (was: 0.5h) > Upgrade com.jayway.jsonpath » json-path to 2.8.0 to fix CVEs > > > Key: HIVE-27290 > URL: https://issues.apache.org/jira/browse/HIVE-27290 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27300) Upgrade to Parquet 1.13.0
[ https://issues.apache.org/jira/browse/HIVE-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko resolved HIVE-27300. --- Resolution: Fixed > Upgrade to Parquet 1.13.0 > - > > Key: HIVE-27300 > URL: https://issues.apache.org/jira/browse/HIVE-27300 > Project: Hive > Issue Type: Improvement > Components: Parquet >Affects Versions: 3.1.3 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27273) Iceberg: Upgrade iceberg to 1.2.1
[
https://issues.apache.org/jira/browse/HIVE-27273?focusedWorklogId=859362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859362
]
ASF GitHub Bot logged work on HIVE-27273:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 09:14
Start Date: 27/Apr/23 09:14
Worklog Time Spent: 10m
Work Description: InvisibleProgrammer commented on PR #4252:
URL: https://github.com/apache/hive/pull/4252#issuecomment-1525221786
@pvary , let me reflect on your writings one by one:
> [*
333227fbd13821365cec1bdbfcb9314a239bea0f](c3232b664745ebf761b6a74f4c5b55cc48bfd209:
Hive: Use EnvironmentContext instead of Hive Locks to provide transactional
commits after HIVE-26882 - This is based on
333227fbd13821365cec1bdbfcb9314a239bea0f and
fede493d59f17ff2bfc0744b296d90bd36130386. Has to be a parallel change on
Hive/Impala and every writers of the Iceberg table, but fixes stability and
enhances commit performance)c3232b664745ebf761b6a74f4c5b55cc48bfd209: Hive: Use
EnvironmentContext instead of Hive Locks to provide transactional commits after
[HIVE-26882](https://issues.apache.org/jira/browse/HIVE-26882) - This is based
on 333227fbd13821365cec1bdbfcb9314a239bea0f and
fede493d59f17ff2bfc0744b296d90bd36130386. Has to be a parallel change on
Hive/Impala and every writers of the Iceberg table, but fixes stability and
enhances commit performance
That is a pretty cool change. I'm pretty sure it is worth porting. But I'm
not sure if we have to port it during the 1.2.1 upgrade. What if I create a
ticket to port it after we finish the 1.2.1 upgrade?
> 333227fbd13821365cec1bdbfcb9314a239bea0f - Hive: Refactor commit lock
mechanism from HiveTableOperations. This is mostly a refactoring to make it
possible to do c3232b664745ebf761b6a74f4c5b55cc48bfd209
That looks trivial, it is easy to port it during this update. Thank your for
the context.
> fede493d59f17ff2bfc0744b296d90bd36130386 - Hive: Lock hardening (#6451) -
makes sure that the Lock used by the Iceberg commit are cleared up... If you do
not have stability issues with stuck Hive Locks then you might skip backporting
it.
That is the tricky and ugly one that concerns me the most: The
`HiveTableOperations` class is almost 1000 lines long in the Hive repository
and about 500 lines in the Iceberg one. And it looks like that commit is the
root cause of the difference. But as you wrote it should be released together
with all writers, it makes it not only ugly but evil as well.
What do you think, even if we have no stability issues with Hive Locks, is
that worth porting?
What do you think, what would be the best way to handle it?
My first thought is the same as at 333227fbd13821365cec1bdbfcb9314a239bea0f:
we should port it in a separated ticket after the 1.2.1 upgrade. But I don't
want to keep that significant difference between the two repositories for a
long time.
And also, do you have any suggestion about how it should be handled in the
community? I mean, I assume we have to start a conversation that includes
Impala and other components as well. I'm not even know, do you know how many
project can be affected?
Issue Time Tracking
---
Worklog Id: (was: 859362)
Time Spent: 2h 20m (was: 2h 10m)
> Iceberg: Upgrade iceberg to 1.2.1
> --
>
> Key: HIVE-27273
> URL: https://issues.apache.org/jira/browse/HIVE-27273
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> [https://iceberg.apache.org/releases/#121-release] Iceberg1.2.1(include
> 1.2.0) has lots of improvement, e.g. _branch commit_ and
> _{{position_deletes}} metadata table._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27300) Upgrade to Parquet 1.13.0
[ https://issues.apache.org/jira/browse/HIVE-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717080#comment-17717080 ] Denys Kuzmenko commented on HIVE-27300: --- Merged to master. [~fokko], thank you for the patch! > Upgrade to Parquet 1.13.0 > - > > Key: HIVE-27300 > URL: https://issues.apache.org/jira/browse/HIVE-27300 > Project: Hive > Issue Type: Improvement > Components: Parquet >Affects Versions: 3.1.3 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27300) Upgrade to Parquet 1.13.0
[ https://issues.apache.org/jira/browse/HIVE-27300?focusedWorklogId=859361&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859361 ] ASF GitHub Bot logged work on HIVE-27300: - Author: ASF GitHub Bot Created on: 27/Apr/23 09:12 Start Date: 27/Apr/23 09:12 Worklog Time Spent: 10m Work Description: deniskuzZ merged PR #4273: URL: https://github.com/apache/hive/pull/4273 Issue Time Tracking --- Worklog Id: (was: 859361) Time Spent: 40m (was: 0.5h) > Upgrade to Parquet 1.13.0 > - > > Key: HIVE-27300 > URL: https://issues.apache.org/jira/browse/HIVE-27300 > Project: Hive > Issue Type: Improvement > Components: Parquet >Affects Versions: 3.1.3 >Reporter: Fokko Driesprong >Assignee: Fokko Driesprong >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27277) Set up github actions workflow to build and push docker image to docker hub
[
https://issues.apache.org/jira/browse/HIVE-27277?focusedWorklogId=859355&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859355
]
ASF GitHub Bot logged work on HIVE-27277:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 08:55
Start Date: 27/Apr/23 08:55
Worklog Time Spent: 10m
Work Description: TuroczyX commented on code in PR #4274:
URL: https://github.com/apache/hive/pull/4274#discussion_r1178833962
##
.github/workflows/docker-image.yml:
##
@@ -0,0 +1,38 @@
+name: ci hive docker image
+
+on:
+ push:
+branches:
+ - "master"
+
+jobs:
+ build:
+runs-on: ubuntu-latest
+steps:
+ -
+name: Checkout
+uses: actions/checkout@v3
+ -
+name: Login to Docker Hub
+uses: docker/login-action@v2
+with:
+ username: ${{ secrets.DOCKERHUB_USERNAME }}
+ password: ${{ secrets.DOCKERHUB_TOKEN }}
+ -
+name: Set up Docker Buildx
+uses: docker/setup-buildx-action@v2
+
+ -
+name: Build and push
+uses: docker/build-push-action@v4
+with:
+ context: ./packaging/src/docker/
+ file: ./packaging/src/docker/Dockerfile
+ push: true
+ tags: ${{ secrets.DOCKERHUB_USERNAME }}/hive:test-image
Review Comment:
How about -hive:dev or -hive:daily?
The GA version should be the same as the industry follows like
hive4.0-latest imho
Issue Time Tracking
---
Worklog Id: (was: 859355)
Time Spent: 1h 10m (was: 1h)
> Set up github actions workflow to build and push docker image to docker hub
> ---
>
> Key: HIVE-27277
> URL: https://issues.apache.org/jira/browse/HIVE-27277
> Project: Hive
> Issue Type: Sub-task
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27277) Set up github actions workflow to build and push docker image to docker hub
[ https://issues.apache.org/jira/browse/HIVE-27277?focusedWorklogId=859354&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859354 ] ASF GitHub Bot logged work on HIVE-27277: - Author: ASF GitHub Bot Created on: 27/Apr/23 08:54 Start Date: 27/Apr/23 08:54 Worklog Time Spent: 10m Work Description: TuroczyX commented on code in PR #4274: URL: https://github.com/apache/hive/pull/4274#discussion_r1178832352 ## .github/workflows/docker-image.yml: ## @@ -0,0 +1,38 @@ +name: ci hive docker image + +on: + push: Review Comment: I think we should have a -latest with the GA version. Also we could have a daily release about the -dev version (or tags) for every commit would be a bit overused. Issue Time Tracking --- Worklog Id: (was: 859354) Time Spent: 1h (was: 50m) > Set up github actions workflow to build and push docker image to docker hub > --- > > Key: HIVE-27277 > URL: https://issues.apache.org/jira/browse/HIVE-27277 > Project: Hive > Issue Type: Sub-task >Reporter: Simhadri Govindappa >Assignee: Simhadri Govindappa >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27186) A persistent property store
[ https://issues.apache.org/jira/browse/HIVE-27186?focusedWorklogId=859351&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859351 ] ASF GitHub Bot logged work on HIVE-27186: - Author: ASF GitHub Bot Created on: 27/Apr/23 08:35 Start Date: 27/Apr/23 08:35 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #4194: URL: https://github.com/apache/hive/pull/4194#issuecomment-1525126670 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4194) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=BUG) [5 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4194&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4194&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4194&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=CODE_SMELL) [104 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4194&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4194&metric=coverage&view=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4194&metric=duplicated_lines_density&view=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 859351) Time Spent: 14h 10m (was: 14h) > A persistent property store > > > Key: HIVE-27186 > URL: https://issues.apache.org/jira/browse/HIVE-27186 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 4.0.0-alpha-2 >Reporter: Henri Biestro >Assignee: Henri Biestro >Priority: Major > Labels: pull-request-available > Time Spent: 14h 10m > Remaining Estimate: 0h > > WHAT > A persistent property store usable as a support facility for any metadata > augmentation feature. > WHY > When adding new meta-data oriented features, we usually need to persist > information linking the feature data and the HiveMetaStore objects it applies > to. Any information related to a database, a table or the cluster - like > statistics for example or any operational data state or data (think rolling > backup) - fall in this use-case. > Typically, accommodating such a feature requires modifying the Metastore > database schema by adding or altering a table.
[jira] [Work logged] (HIVE-27163) Column stats are not getting published after an insert query into an external table with custom location
[
https://issues.apache.org/jira/browse/HIVE-27163?focusedWorklogId=859336&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859336
]
ASF GitHub Bot logged work on HIVE-27163:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 07:50
Start Date: 27/Apr/23 07:50
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #4228:
URL: https://github.com/apache/hive/pull/4228#issuecomment-1525017849
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4228)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=BUG)
[1
Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4228&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4228&resolved=false&types=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4228&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=CODE_SMELL)
[18 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4228&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4228&metric=coverage&view=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4228&metric=duplicated_lines_density&view=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 859336)
Time Spent: 2.5h (was: 2h 20m)
> Column stats are not getting published after an insert query into an external
> table with custom location
>
>
> Key: HIVE-27163
> URL: https://issues.apache.org/jira/browse/HIVE-27163
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Test case details are below
> *test.q*
> {noformat}
> set hive.stats.column.autogather=true;
> set hive.stats.autogather=true;
> dfs ${system:test.dfs.mkdir} ${system:test.tmp.dir}/test;
> create external table test_custom(age int, name string) stored as orc
> location '/tmp/test';
> insert into test_custom select 1, 'test';
> desc formatted test_custom age;{noformat}
> *test.q.out*
>
>
> {noformat}
> A masked pattern was here
> PREHOOK: type: CREATETA
[jira] [Resolved] (HIVE-27295) Improve docker logging in AbstractExternalDB and DatabaseRule
[
https://issues.apache.org/jira/browse/HIVE-27295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor resolved HIVE-27295.
-
Resolution: Fixed
> Improve docker logging in AbstractExternalDB and DatabaseRule
> -
>
> Key: HIVE-27295
> URL: https://issues.apache.org/jira/browse/HIVE-27295
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> 1. While waiting for docker container to start properly, we should print the
> output of docker logs command in every loop, otherwise we can miss important
> information about the actual startup process if the docker container was oom
> killed in the meantime. Not to mention the fact that we're currently not
> logging the output at all in case of an error:
> https://github.com/apache/hive/blob/59058c65457fb7ab9d8575a555034e6633962661/itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java#L125-L127
> 2. We can include the output for docker events in the logs in case of an
> error (like: oom killed container), which might contain useful information.
> We can have info like this:
> {code}
> 2023-04-25T08:47:08.852515314-07:00 container oom
> 2ba12cd9cd844bb30b3158564bd68cd97f25e7a05172d111713ac9f7c1c0b1d4
> (image=harbor.rke-us-west-04.kc.cloudera.com/docker_private_cache/cloudera_thirdparty/postgres:9.3,
> name=qtestExternalDB-PostgresExternalDB)
> 2023-04-25T08:47:08.893742200-07:00 container die
> 2ba12cd9cd844bb30b3158564bd68cd97f25e7a05172d111713ac9f7c1c0b1d4 (exitCode=1,
> image=harbor.rke-us-west-04.kc.cloudera.com/docker_private_cache/cloudera_thirdparty/postgres:9.3,
> name=qtestExternalDB-PostgresExternalDB)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27295) Improve docker logging in AbstractExternalDB and DatabaseRule
[
https://issues.apache.org/jira/browse/HIVE-27295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-27295:
Fix Version/s: 4.0.0
> Improve docker logging in AbstractExternalDB and DatabaseRule
> -
>
> Key: HIVE-27295
> URL: https://issues.apache.org/jira/browse/HIVE-27295
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> 1. While waiting for docker container to start properly, we should print the
> output of docker logs command in every loop, otherwise we can miss important
> information about the actual startup process if the docker container was oom
> killed in the meantime. Not to mention the fact that we're currently not
> logging the output at all in case of an error:
> https://github.com/apache/hive/blob/59058c65457fb7ab9d8575a555034e6633962661/itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java#L125-L127
> 2. We can include the output for docker events in the logs in case of an
> error (like: oom killed container), which might contain useful information.
> We can have info like this:
> {code}
> 2023-04-25T08:47:08.852515314-07:00 container oom
> 2ba12cd9cd844bb30b3158564bd68cd97f25e7a05172d111713ac9f7c1c0b1d4
> (image=harbor.rke-us-west-04.kc.cloudera.com/docker_private_cache/cloudera_thirdparty/postgres:9.3,
> name=qtestExternalDB-PostgresExternalDB)
> 2023-04-25T08:47:08.893742200-07:00 container die
> 2ba12cd9cd844bb30b3158564bd68cd97f25e7a05172d111713ac9f7c1c0b1d4 (exitCode=1,
> image=harbor.rke-us-west-04.kc.cloudera.com/docker_private_cache/cloudera_thirdparty/postgres:9.3,
> name=qtestExternalDB-PostgresExternalDB)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-27295) Improve docker logging in AbstractExternalDB and DatabaseRule
[
https://issues.apache.org/jira/browse/HIVE-27295?focusedWorklogId=859330&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-859330
]
ASF GitHub Bot logged work on HIVE-27295:
-
Author: ASF GitHub Bot
Created on: 27/Apr/23 07:08
Start Date: 27/Apr/23 07:08
Worklog Time Spent: 10m
Work Description: abstractdog merged PR #4268:
URL: https://github.com/apache/hive/pull/4268
Issue Time Tracking
---
Worklog Id: (was: 859330)
Time Spent: 2h 10m (was: 2h)
> Improve docker logging in AbstractExternalDB and DatabaseRule
> -
>
> Key: HIVE-27295
> URL: https://issues.apache.org/jira/browse/HIVE-27295
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> 1. While waiting for docker container to start properly, we should print the
> output of docker logs command in every loop, otherwise we can miss important
> information about the actual startup process if the docker container was oom
> killed in the meantime. Not to mention the fact that we're currently not
> logging the output at all in case of an error:
> https://github.com/apache/hive/blob/59058c65457fb7ab9d8575a555034e6633962661/itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java#L125-L127
> 2. We can include the output for docker events in the logs in case of an
> error (like: oom killed container), which might contain useful information.
> We can have info like this:
> {code}
> 2023-04-25T08:47:08.852515314-07:00 container oom
> 2ba12cd9cd844bb30b3158564bd68cd97f25e7a05172d111713ac9f7c1c0b1d4
> (image=harbor.rke-us-west-04.kc.cloudera.com/docker_private_cache/cloudera_thirdparty/postgres:9.3,
> name=qtestExternalDB-PostgresExternalDB)
> 2023-04-25T08:47:08.893742200-07:00 container die
> 2ba12cd9cd844bb30b3158564bd68cd97f25e7a05172d111713ac9f7c1c0b1d4 (exitCode=1,
> image=harbor.rke-us-west-04.kc.cloudera.com/docker_private_cache/cloudera_thirdparty/postgres:9.3,
> name=qtestExternalDB-PostgresExternalDB)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27295) Improve docker logging in AbstractExternalDB and DatabaseRule
[
https://issues.apache.org/jira/browse/HIVE-27295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717021#comment-17717021
]
László Bodor commented on HIVE-27295:
-
merged to master, thanks [~zabetak] for the review!
> Improve docker logging in AbstractExternalDB and DatabaseRule
> -
>
> Key: HIVE-27295
> URL: https://issues.apache.org/jira/browse/HIVE-27295
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> 1. While waiting for docker container to start properly, we should print the
> output of docker logs command in every loop, otherwise we can miss important
> information about the actual startup process if the docker container was oom
> killed in the meantime. Not to mention the fact that we're currently not
> logging the output at all in case of an error:
> https://github.com/apache/hive/blob/59058c65457fb7ab9d8575a555034e6633962661/itests/util/src/main/java/org/apache/hadoop/hive/ql/externalDB/AbstractExternalDB.java#L125-L127
> 2. We can include the output for docker events in the logs in case of an
> error (like: oom killed container), which might contain useful information.
> We can have info like this:
> {code}
> 2023-04-25T08:47:08.852515314-07:00 container oom
> 2ba12cd9cd844bb30b3158564bd68cd97f25e7a05172d111713ac9f7c1c0b1d4
> (image=harbor.rke-us-west-04.kc.cloudera.com/docker_private_cache/cloudera_thirdparty/postgres:9.3,
> name=qtestExternalDB-PostgresExternalDB)
> 2023-04-25T08:47:08.893742200-07:00 container die
> 2ba12cd9cd844bb30b3158564bd68cd97f25e7a05172d111713ac9f7c1c0b1d4 (exitCode=1,
> image=harbor.rke-us-west-04.kc.cloudera.com/docker_private_cache/cloudera_thirdparty/postgres:9.3,
> name=qtestExternalDB-PostgresExternalDB)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
