[jira] [Work logged] (HIVE-26980) Creation of materialized view stored by Iceberg fails if source table has tinyint column
[
https://issues.apache.org/jira/browse/HIVE-26980?focusedWorklogId=842450=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842450
]
ASF GitHub Bot logged work on HIVE-26980:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 07:57
Start Date: 31/Jan/23 07:57
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3993:
URL: https://github.com/apache/hive/pull/3993#issuecomment-1409920093
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3993)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3993=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3993=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3993=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3993=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3993=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 842450)
Time Spent: 2h (was: 1h 50m)
> Creation of materialized view stored by Iceberg fails if source table has
> tinyint column
>
>
> Key: HIVE-26980
> URL: https://issues.apache.org/jira/browse/HIVE-26980
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> MV creation using iceberg storage format for Hive table containing tinyint
> column fails with error "java.lang.ClassCastException: class java.lang.Byte
> cannot be cast to class java.lang.Integer"
> Logs
> {code:java}
> 2023-01-24T21:36:22,153 INFO [pool-2-thread-1] jdbc.TestDriver: Beginning
> Test at 2023-01-24 21:36:22,153
> 2023-01-24T21:36:22,154 INFO [pool-2-thread-1] jdbc.TestDriver: BEGIN MAIN
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Running
> SessionGroup{name='SG_TNMECIDQ31', initialDelay=0, repeats=1, repeatDelay=0}
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Connecting
> as user 'hrt_qa'
> 2023-01-24T21:36:22,409 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.cbo.enable=true
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:22,619
[jira] [Work logged] (HIVE-26804) Cancel Compactions in initiated state
[
https://issues.apache.org/jira/browse/HIVE-26804?focusedWorklogId=842447=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842447
]
ASF GitHub Bot logged work on HIVE-26804:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 07:50
Start Date: 31/Jan/23 07:50
Worklog Time Spent: 10m
Work Description: rkirtir commented on code in PR #3880:
URL: https://github.com/apache/hive/pull/3880#discussion_r1091558532
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -6242,4 +6245,91 @@ public boolean isWrapperFor(Class iface) throws
SQLException {
}
}
+ @Override
+ @RetrySemantics.SafeToRetry
+ public AbortCompactResponse abortCompactions(AbortCompactionRequest reqst)
throws MetaException, NoSuchCompactionException {
+
+AbortCompactResponse response = new AbortCompactResponse(new HashMap<>());
+response.setAbortedcompacts(abortCompactionResponseElements);
+List compactionIdsToAbort = reqst.getCompactionIds();
+if (compactionIdsToAbort.isEmpty()) {
+ LOG.info("Compaction ids are missing in request. No compactions to
abort");
+ throw new NoSuchCompactionException("Compaction ids missing in request.
No compactions to abort");
+}
+reqst.getCompactionIds().forEach(x -> {
+ abortCompactionResponseElements.put(x, new
AbortCompactionResponseElement(x, "Error", "Not Eligible"));
Review Comment:
fixed
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -6266,4 +6271,105 @@ public boolean isWrapperFor(Class iface) throws
SQLException {
}
}
+ @Override
+ @RetrySemantics.SafeToRetry
+ public AbortCompactResponse abortCompactions(AbortCompactionRequest reqst)
throws MetaException, NoSuchCompactionException {
+Map abortCompactionResponseElements
= new HashMap<>();
+AbortCompactResponse response = new AbortCompactResponse(new HashMap<>());
+response.setAbortedcompacts(abortCompactionResponseElements);
+
+List compactionIdsToAbort = reqst.getCompactionIds();
+if (compactionIdsToAbort.isEmpty()) {
+ LOG.info("Compaction ids are missing in request. No compactions to
abort");
+ throw new NoSuchCompactionException("Compaction ids missing in request.
No compactions to abort");
+}
+reqst.getCompactionIds().forEach(x -> {
+ abortCompactionResponseElements.put(x, new
AbortCompactionResponseElement(x, "Error",
+ "No Such Compaction Id Available"));
+});
+
+List eligibleCompactionsToAbort =
findEligibleCompactionsToAbort(abortCompactionResponseElements,
+compactionIdsToAbort);
+for (int x = 0; x < eligibleCompactionsToAbort.size(); x++) {
Review Comment:
fixed
Issue Time Tracking
---
Worklog Id: (was: 842447)
Time Spent: 5h 10m (was: 5h)
> Cancel Compactions in initiated state
> -
>
> Key: HIVE-26804
> URL: https://issues.apache.org/jira/browse/HIVE-26804
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26804) Cancel Compactions in initiated state
[
https://issues.apache.org/jira/browse/HIVE-26804?focusedWorklogId=842448=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842448
]
ASF GitHub Bot logged work on HIVE-26804:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 07:50
Start Date: 31/Jan/23 07:50
Worklog Time Spent: 10m
Work Description: rkirtir commented on code in PR #3880:
URL: https://github.com/apache/hive/pull/3880#discussion_r1091558840
##
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnQueries.java:
##
@@ -18,36 +18,70 @@
package org.apache.hadoop.hive.metastore.txn;
public class TxnQueries {
- public static final String SHOW_COMPACTION_ORDERBY_CLAUSE =
-" ORDER BY CASE " +
-" WHEN \"CC_END\" > \"CC_START\" and \"CC_END\" > \"CC_COMMIT_TIME\" " +
-" THEN \"CC_END\" " +
-" WHEN \"CC_START\" > \"CC_COMMIT_TIME\" " +
-" THEN \"CC_START\" " +
-" ELSE \"CC_COMMIT_TIME\" " +
-" END desc ," +
-" \"CC_ENQUEUE_TIME\" asc";
+public static final String SHOW_COMPACTION_ORDERBY_CLAUSE =
+" ORDER BY CASE " +
+" WHEN \"CC_END\" > \"CC_START\" and \"CC_END\" >
\"CC_COMMIT_TIME\" " +
+" THEN \"CC_END\" " +
+" WHEN \"CC_START\" > \"CC_COMMIT_TIME\" " +
+" THEN \"CC_START\" " +
+" ELSE \"CC_COMMIT_TIME\" " +
+" END desc ," +
+" \"CC_ENQUEUE_TIME\" asc";
- public static final String SHOW_COMPACTION_QUERY =
-"SELECT XX.* FROM ( SELECT " +
-" \"CQ_DATABASE\" AS \"CC_DATABASE\", \"CQ_TABLE\" AS \"CC_TABLE\",
\"CQ_PARTITION\" AS \"CC_PARTITION\", " +
-" \"CQ_STATE\" AS \"CC_STATE\", \"CQ_TYPE\" AS \"CC_TYPE\",
\"CQ_WORKER_ID\" AS \"CC_WORKER_ID\", " +
-" \"CQ_START\" AS \"CC_START\", -1 \"CC_END\", \"CQ_RUN_AS\" AS
\"CC_RUN_AS\", " +
-" \"CQ_HADOOP_JOB_ID\" AS \"CC_HADOOP_JOB_ID\", \"CQ_ID\" AS \"CC_ID\",
\"CQ_ERROR_MESSAGE\" AS \"CC_ERROR_MESSAGE\", " +
-" \"CQ_ENQUEUE_TIME\" AS \"CC_ENQUEUE_TIME\", \"CQ_WORKER_VERSION\" AS
\"CC_WORKER_VERSION\", " +
-" \"CQ_INITIATOR_ID\" AS \"CC_INITIATOR_ID\", \"CQ_INITIATOR_VERSION\" AS
\"CC_INITIATOR_VERSION\", " +
-" \"CQ_CLEANER_START\" AS \"CC_CLEANER_START\", \"CQ_POOL_NAME\" AS
\"CC_POOL_NAME\", \"CQ_TXN_ID\" AS \"CC_TXN_ID\", " +
-" \"CQ_NEXT_TXN_ID\" AS \"CC_NEXT_TXN_ID\", \"CQ_COMMIT_TIME\" AS
\"CC_COMMIT_TIME\", " +
-" \"CQ_HIGHEST_WRITE_ID\" AS \"CC_HIGHEST_WRITE_ID\" " +
-"FROM " +
-" \"COMPACTION_QUEUE\" " +
-"UNION ALL " +
-"SELECT " +
-" \"CC_DATABASE\", \"CC_TABLE\", \"CC_PARTITION\", \"CC_STATE\",
\"CC_TYPE\", \"CC_WORKER_ID\", " +
-" \"CC_START\", \"CC_END\", \"CC_RUN_AS\", \"CC_HADOOP_JOB_ID\",
\"CC_ID\", \"CC_ERROR_MESSAGE\", " +
-" \"CC_ENQUEUE_TIME\", \"CC_WORKER_VERSION\", \"CC_INITIATOR_ID\",
\"CC_INITIATOR_VERSION\", " +
-" -1 , \"CC_POOL_NAME\", \"CC_TXN_ID\", \"CC_NEXT_TXN_ID\",
\"CC_COMMIT_TIME\", " +
-" \"CC_HIGHEST_WRITE_ID\"" +
-"FROM " +
-" \"COMPLETED_COMPACTIONS\" ) XX ";
+public static final String SHOW_COMPACTION_QUERY =
+"SELECT XX.* FROM ( SELECT " +
+" \"CQ_DATABASE\" AS \"CC_DATABASE\", \"CQ_TABLE\" AS
\"CC_TABLE\", \"CQ_PARTITION\" AS \"CC_PARTITION\", " +
+" \"CQ_STATE\" AS \"CC_STATE\", \"CQ_TYPE\" AS
\"CC_TYPE\", \"CQ_WORKER_ID\" AS \"CC_WORKER_ID\", " +
+" \"CQ_START\" AS \"CC_START\", -1 \"CC_END\",
\"CQ_RUN_AS\" AS \"CC_RUN_AS\", " +
+" \"CQ_HADOOP_JOB_ID\" AS \"CC_HADOOP_JOB_ID\", \"CQ_ID\"
AS \"CC_ID\", \"CQ_ERROR_MESSAGE\" AS \"CC_ERROR_MESSAGE\", " +
+" \"CQ_ENQUEUE_TIME\" AS \"CC_ENQUEUE_TIME\",
\"CQ_WORKER_VERSION\" AS \"CC_WORKER_VERSION\", " +
+" \"CQ_INITIATOR_ID\" AS \"CC_INITIATOR_ID\",
\"CQ_INITIATOR_VERSION\" AS \"CC_INITIATOR_VERSION\", " +
+" \"CQ_CLEANER_START\" AS \"CC_CLEANER_START\",
\"CQ_POOL_NAME\" AS \"CC_POOL_NAME\", \"CQ_TXN_ID\" AS \"CC_TXN_ID\", " +
+" \"CQ_NEXT_TXN_ID\" AS \"CC_NEXT_TXN_ID\",
\"CQ_COMMIT_TIME\" AS \"CC_COMMIT_TIME\", " +
+" \"CQ_HIGHEST_WRITE_ID\" AS \"CC_HIGHEST_WRITE_ID\" " +
+"FROM " +
+" \"COMPACTION_QUEUE\" " +
+"UNION ALL " +
+"SELECT " +
+" \"CC_DATABASE\" , \"CC_TABLE\", \"CC_PARTITION\",
\"CC_STATE\", \"CC_TYPE\", \"CC_WORKER_ID\", " +
+" \"CC_START\", \"CC_END\", \"CC_RUN_AS\",
\"CC_HADOOP_JOB_ID\", \"CC_ID\", \"CC_ERROR_MESSAGE\", " +
+" \"CC_ENQUEUE_TIME\", \"CC_WORKER_VERSION\",
\"CC_INITIATOR_ID\", \"CC_INITIATOR_VERSION\", " +
+"
[jira] [Work logged] (HIVE-26804) Cancel Compactions in initiated state
[
https://issues.apache.org/jira/browse/HIVE-26804?focusedWorklogId=842446=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842446
]
ASF GitHub Bot logged work on HIVE-26804:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 07:50
Start Date: 31/Jan/23 07:50
Worklog Time Spent: 10m
Work Description: rkirtir commented on code in PR #3880:
URL: https://github.com/apache/hive/pull/3880#discussion_r1091558263
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java:
##
@@ -6242,4 +6246,86 @@ public boolean isWrapperFor(Class iface) throws
SQLException {
}
}
+ @Override
+ @RetrySemantics.SafeToRetry
+ public AbortCompactResponse abortCompactions(AbortCompactionRequest reqst)
throws MetaException, NoSuchCompactionException {
Review Comment:
fixed
Issue Time Tracking
---
Worklog Id: (was: 842446)
Time Spent: 5h (was: 4h 50m)
> Cancel Compactions in initiated state
> -
>
> Key: HIVE-26804
> URL: https://issues.apache.org/jira/browse/HIVE-26804
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: KIRTI RUGE
>Assignee: KIRTI RUGE
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work started] (HIVE-26793) Create a new configuration to override "no compaction" for tables
[
https://issues.apache.org/jira/browse/HIVE-26793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-26793 started by Kokila N.
---
> Create a new configuration to override "no compaction" for tables
> -
>
> Key: HIVE-26793
> URL: https://issues.apache.org/jira/browse/HIVE-26793
> Project: Hive
> Issue Type: Improvement
>Reporter: Kokila N
>Assignee: Kokila N
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Currently a simple user can create a table with
> {color:#6a8759}no_auto_compaction=true{color} table property and create an
> aborted write transaction writing to this table. This way a malicious user
> can prevent cleaning up data for the aborted transaction, creating
> performance degradation.
> This configuration should be allowed to overridden on a database level:
> adding {color:#6a8759}no_auto_compaction=false{color} should override the
> table level setting forcing the initiator to schedule compaction for all
> tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26980) Creation of materialized view stored by Iceberg fails if source table has tinyint column
[
https://issues.apache.org/jira/browse/HIVE-26980?focusedWorklogId=842441=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842441
]
ASF GitHub Bot logged work on HIVE-26980:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 07:32
Start Date: 31/Jan/23 07:32
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3993:
URL: https://github.com/apache/hive/pull/3993#discussion_r1091544722
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7878,6 +7878,10 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
throw new SemanticException("Unknown destination type: " + destType);
}
+if (!(destType == QBMetaData.DEST_DFS_FILE && qb.getIsQuery())) {
Review Comment:
Issue Time Tracking
---
Worklog Id: (was: 842441)
Time Spent: 1h 50m (was: 1h 40m)
> Creation of materialized view stored by Iceberg fails if source table has
> tinyint column
>
>
> Key: HIVE-26980
> URL: https://issues.apache.org/jira/browse/HIVE-26980
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> MV creation using iceberg storage format for Hive table containing tinyint
> column fails with error "java.lang.ClassCastException: class java.lang.Byte
> cannot be cast to class java.lang.Integer"
> Logs
> {code:java}
> 2023-01-24T21:36:22,153 INFO [pool-2-thread-1] jdbc.TestDriver: Beginning
> Test at 2023-01-24 21:36:22,153
> 2023-01-24T21:36:22,154 INFO [pool-2-thread-1] jdbc.TestDriver: BEGIN MAIN
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Running
> SessionGroup{name='SG_TNMECIDQ31', initialDelay=0, repeats=1, repeatDelay=0}
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Connecting
> as user 'hrt_qa'
> 2023-01-24T21:36:22,409 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.cbo.enable=true
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.materializedview.rewriting
> 2023-01-24T21:36:22,946 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> iceberg.mr.schema.auto.conversion=true
> 2023-01-24T21:36:23,183 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:23,183 INFO [pool-38-thread-1] jdbc.TestDriver: Query: drop
> materialized view if exists mv_project
> 2023-01-24T21:36:23,243 INFO [Thread-401] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3):
> drop materialized view if exists mv_project
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Semantic
> Analysis Completed (retrial = false)
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Created
> Hive schema: Schema(fieldSchemas:null, properties:null)
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Completed
> compiling
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3);
> Time taken: 0.03 seconds
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Executing
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3):
> drop materialized view if exists mv_project
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Starting
> task [Stage-0:DDL] in serial mode
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Completed
> executing
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3);
> Time taken: 0.008 seconds
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : OK
> 2023-01-24T21:36:23,423 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:23,423 INFO [pool-38-thread-1] jdbc.TestDriver: Query:
> create materialized view mv_project stored by iceberg as select t, si, avg(t)
> from all100k where t>115 group by t, si
> 2023-01-24T21:36:24,015 INFO [Thread-402] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230124213623_e6e322e5-fa9b-4fb2-a3cb-ae244699ee5d):
> create materialized view mv_project stored by iceberg as select t, si, avg(t)
> from all100k where t>115 group by t, si
> 2023-01-24T21:36:24,015 INFO [Thread-402] jdbc.TestDriver: INFO : Semantic
> Analysis Completed (retrial = false)
> 2023-01-24T21:36:24,015 INFO [Thread-402] jdbc.TestDriver: INFO : Created
> Hive schema:
[jira] [Work logged] (HIVE-27000) Improve the modularity of the *ColumnStatsMerger classes
[ https://issues.apache.org/jira/browse/HIVE-27000?focusedWorklogId=842437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842437 ] ASF GitHub Bot logged work on HIVE-27000: - Author: ASF GitHub Bot Created on: 31/Jan/23 07:22 Start Date: 31/Jan/23 07:22 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3997: URL: https://github.com/apache/hive/pull/3997#issuecomment-1409885334 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3997) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3997=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3997=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3997=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3997=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3997=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842437) Time Spent: 0.5h (was: 20m) > Improve the modularity of the *ColumnStatsMerger classes > > > Key: HIVE-27000 > URL: https://issues.apache.org/jira/browse/HIVE-27000 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > *ColumnStatsMerger classes contain a lot of duplicate code which is not > specific to the data type, and that could therefore be lifted to a common > parent class. > This phenomenon is bound to become even worse if we keep enriching further > our supported set of statistics as we did in the context of HIVE-26221. > The current ticket aims at improving the modularity and code reuse of the > *ColumnStatsMerger classes, while improving unit-test coverage to cover all > classes and support more use-cases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26980) Creation of materialized view stored by Iceberg fails if source table has tinyint column
[
https://issues.apache.org/jira/browse/HIVE-26980?focusedWorklogId=842435=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842435
]
ASF GitHub Bot logged work on HIVE-26980:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 07:16
Start Date: 31/Jan/23 07:16
Worklog Time Spent: 10m
Work Description: InvisibleProgrammer commented on PR #3993:
URL: https://github.com/apache/hive/pull/3993#issuecomment-1409880571
> The only difference is that you explicitly enabled vectorization but that
is enabled by default in PTests.
Sad to hear. That means, if we run the test on the build server, we get a
different result than running it on our machine. Could you please add that
vectorization test to the setting to get the exact same behavior?
Issue Time Tracking
---
Worklog Id: (was: 842435)
Time Spent: 1h 40m (was: 1.5h)
> Creation of materialized view stored by Iceberg fails if source table has
> tinyint column
>
>
> Key: HIVE-26980
> URL: https://issues.apache.org/jira/browse/HIVE-26980
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> MV creation using iceberg storage format for Hive table containing tinyint
> column fails with error "java.lang.ClassCastException: class java.lang.Byte
> cannot be cast to class java.lang.Integer"
> Logs
> {code:java}
> 2023-01-24T21:36:22,153 INFO [pool-2-thread-1] jdbc.TestDriver: Beginning
> Test at 2023-01-24 21:36:22,153
> 2023-01-24T21:36:22,154 INFO [pool-2-thread-1] jdbc.TestDriver: BEGIN MAIN
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Running
> SessionGroup{name='SG_TNMECIDQ31', initialDelay=0, repeats=1, repeatDelay=0}
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Connecting
> as user 'hrt_qa'
> 2023-01-24T21:36:22,409 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.cbo.enable=true
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.materializedview.rewriting
> 2023-01-24T21:36:22,946 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> iceberg.mr.schema.auto.conversion=true
> 2023-01-24T21:36:23,183 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:23,183 INFO [pool-38-thread-1] jdbc.TestDriver: Query: drop
> materialized view if exists mv_project
> 2023-01-24T21:36:23,243 INFO [Thread-401] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3):
> drop materialized view if exists mv_project
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Semantic
> Analysis Completed (retrial = false)
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Created
> Hive schema: Schema(fieldSchemas:null, properties:null)
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Completed
> compiling
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3);
> Time taken: 0.03 seconds
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Executing
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3):
> drop materialized view if exists mv_project
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Starting
> task [Stage-0:DDL] in serial mode
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Completed
> executing
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3);
> Time taken: 0.008 seconds
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : OK
> 2023-01-24T21:36:23,423 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:23,423 INFO [pool-38-thread-1] jdbc.TestDriver: Query:
> create materialized view mv_project stored by iceberg as select t, si, avg(t)
> from all100k where t>115 group by t, si
> 2023-01-24T21:36:24,015 INFO [Thread-402] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230124213623_e6e322e5-fa9b-4fb2-a3cb-ae244699ee5d):
> create materialized view mv_project stored by iceberg as select t, si, avg(t)
> from all100k where t>115 group by t, si
> 2023-01-24T21:36:24,015 INFO [Thread-402] jdbc.TestDriver: INFO : Semantic
> Analysis Completed (retrial = false)
> 2023-01-24T21:36:24,015 INFO [Thread-402] jdbc.TestDriver: INFO : Created
> Hive schema: Schema(fieldSchemas:[FieldSchema(name:t, type:tinyint,
[jira] [Work logged] (HIVE-26980) Creation of materialized view stored by Iceberg fails if source table has tinyint column
[
https://issues.apache.org/jira/browse/HIVE-26980?focusedWorklogId=842434=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842434
]
ASF GitHub Bot logged work on HIVE-26980:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 07:08
Start Date: 31/Jan/23 07:08
Worklog Time Spent: 10m
Work Description: kasakrisz commented on PR #3993:
URL: https://github.com/apache/hive/pull/3993#issuecomment-1409868744
@InvisibleProgrammer
Thanks for validating this patch. I checked your test case and found that
the test `iceberg/iceberg-handler/src/test/queries/positive/ctas_iceberg_orc.q`
updated by this patch covers your scenario. The only difference is that you
explicitly enabled vectorization but that is enabled by default in PTests.
Issue Time Tracking
---
Worklog Id: (was: 842434)
Time Spent: 1.5h (was: 1h 20m)
> Creation of materialized view stored by Iceberg fails if source table has
> tinyint column
>
>
> Key: HIVE-26980
> URL: https://issues.apache.org/jira/browse/HIVE-26980
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> MV creation using iceberg storage format for Hive table containing tinyint
> column fails with error "java.lang.ClassCastException: class java.lang.Byte
> cannot be cast to class java.lang.Integer"
> Logs
> {code:java}
> 2023-01-24T21:36:22,153 INFO [pool-2-thread-1] jdbc.TestDriver: Beginning
> Test at 2023-01-24 21:36:22,153
> 2023-01-24T21:36:22,154 INFO [pool-2-thread-1] jdbc.TestDriver: BEGIN MAIN
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Running
> SessionGroup{name='SG_TNMECIDQ31', initialDelay=0, repeats=1, repeatDelay=0}
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Connecting
> as user 'hrt_qa'
> 2023-01-24T21:36:22,409 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.cbo.enable=true
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.materializedview.rewriting
> 2023-01-24T21:36:22,946 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> iceberg.mr.schema.auto.conversion=true
> 2023-01-24T21:36:23,183 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:23,183 INFO [pool-38-thread-1] jdbc.TestDriver: Query: drop
> materialized view if exists mv_project
> 2023-01-24T21:36:23,243 INFO [Thread-401] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3):
> drop materialized view if exists mv_project
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Semantic
> Analysis Completed (retrial = false)
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Created
> Hive schema: Schema(fieldSchemas:null, properties:null)
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Completed
> compiling
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3);
> Time taken: 0.03 seconds
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Executing
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3):
> drop materialized view if exists mv_project
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Starting
> task [Stage-0:DDL] in serial mode
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Completed
> executing
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3);
> Time taken: 0.008 seconds
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : OK
> 2023-01-24T21:36:23,423 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:23,423 INFO [pool-38-thread-1] jdbc.TestDriver: Query:
> create materialized view mv_project stored by iceberg as select t, si, avg(t)
> from all100k where t>115 group by t, si
> 2023-01-24T21:36:24,015 INFO [Thread-402] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230124213623_e6e322e5-fa9b-4fb2-a3cb-ae244699ee5d):
> create materialized view mv_project stored by iceberg as select t, si, avg(t)
> from all100k where t>115 group by t, si
> 2023-01-24T21:36:24,015 INFO [Thread-402] jdbc.TestDriver: INFO : Semantic
> Analysis Completed (retrial = false)
> 2023-01-24T21:36:24,015 INFO [Thread-402] jdbc.TestDriver: INFO : Created
> Hive schema: Schema(fieldSchemas:[FieldSchema(name:t, type:tinyint,
>
[jira] [Work logged] (HIVE-26980) Creation of materialized view stored by Iceberg fails if source table has tinyint column
[
https://issues.apache.org/jira/browse/HIVE-26980?focusedWorklogId=842432=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842432
]
ASF GitHub Bot logged work on HIVE-26980:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 07:02
Start Date: 31/Jan/23 07:02
Worklog Time Spent: 10m
Work Description: kasakrisz commented on code in PR #3993:
URL: https://github.com/apache/hive/pull/3993#discussion_r1091524011
##
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##
@@ -7878,6 +7878,10 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
throw new SemanticException("Unknown destination type: " + destType);
}
+if (!(destType == QBMetaData.DEST_DFS_FILE && qb.getIsQuery())) {
Review Comment:
* IIUC when destType=DEST_LOCAL_FILE we are inserting into a local directory
and no conversion is required since the target is not an actual table. Example.
https://github.com/apache/hive/blob/0a5558d71c474f285c1e6d338a4be93bd1c6ce13/ql/src/test/queries/clientpositive/insert_overwrite_local_directory_1.q#L2
I extended the if condition with checking whether the destinationTable is
null and does it has a storage handler.
* yes, isQuery should be false for ctas and mv
https://github.com/apache/hive/blob/0a5558d71c474f285c1e6d338a4be93bd1c6ce13/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L2443-L2444
Issue Time Tracking
---
Worklog Id: (was: 842432)
Time Spent: 1h 20m (was: 1h 10m)
> Creation of materialized view stored by Iceberg fails if source table has
> tinyint column
>
>
> Key: HIVE-26980
> URL: https://issues.apache.org/jira/browse/HIVE-26980
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> MV creation using iceberg storage format for Hive table containing tinyint
> column fails with error "java.lang.ClassCastException: class java.lang.Byte
> cannot be cast to class java.lang.Integer"
> Logs
> {code:java}
> 2023-01-24T21:36:22,153 INFO [pool-2-thread-1] jdbc.TestDriver: Beginning
> Test at 2023-01-24 21:36:22,153
> 2023-01-24T21:36:22,154 INFO [pool-2-thread-1] jdbc.TestDriver: BEGIN MAIN
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Running
> SessionGroup{name='SG_TNMECIDQ31', initialDelay=0, repeats=1, repeatDelay=0}
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Connecting
> as user 'hrt_qa'
> 2023-01-24T21:36:22,409 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.cbo.enable=true
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.materializedview.rewriting
> 2023-01-24T21:36:22,946 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> iceberg.mr.schema.auto.conversion=true
> 2023-01-24T21:36:23,183 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:23,183 INFO [pool-38-thread-1] jdbc.TestDriver: Query: drop
> materialized view if exists mv_project
> 2023-01-24T21:36:23,243 INFO [Thread-401] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3):
> drop materialized view if exists mv_project
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Semantic
> Analysis Completed (retrial = false)
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Created
> Hive schema: Schema(fieldSchemas:null, properties:null)
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Completed
> compiling
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3);
> Time taken: 0.03 seconds
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Executing
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3):
> drop materialized view if exists mv_project
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Starting
> task [Stage-0:DDL] in serial mode
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Completed
> executing
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3);
> Time taken: 0.008 seconds
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : OK
> 2023-01-24T21:36:23,423 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:23,423 INFO [pool-38-thread-1] jdbc.TestDriver: Query:
>
[jira] [Updated] (HIVE-27004) DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
[
https://issues.apache.org/jira/browse/HIVE-27004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anmol Sundaram updated HIVE-27004:
--
Description:
Some of the unit tests related to _DateTimeFormatter_ were failing in Java
versions greater than 8 while working in Java 8.
Example of a failing Unit Test :
_org.apache.hadoop.hive.common.type.TestTimestampTZ#testComparision_
{code:java}
java.time.format.DateTimeParseException: Text '2017-04-14 18:00:00 UTC+08:00'
could not be parsed, unparsed text found at index 23 at
java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
at
java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
at
org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:76)
at
org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:64)
at
org.apache.hadoop.hive.common.type.TestTimestampTZ.testComparison(TestTimestampTZ.java:44)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method){code}
The {{appendZoneText(TextStyle)}} method of
[DateTimeFormatteBuilder|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java#L82]
is not able to parse the {{+}} symbol in {{2017-04-14 18:00:00 UTC+08:00}}
when running [the
test|https://github.com/apache/hive/blob/master/common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java#L37]
in Java 11 , while it is working fine with Java 8.
According to the
[doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendZoneText(java.time.format.TextStyle)]
, {{appendZoneText(TextStyle)}} should be able to parse {{either the textual
zone name, the zone ID or the offset}} and UTC+08:00 should come under offset
as per the same
[doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendOffset(java.lang.String,%20java.lang.String)].
It seems, however, that "UTC" was explicitly removed [when parsed as ZoneText
for higher Java
versions|https://github.com/openjdk/jdk/commit/5c3a01591c5c945926636fdc9f164d60b5b4f29e?diff=unified#diff-5fcf976db1c06e8f44a8671356d7e34fdfbf5b057baa852e7c3e015c8797c889R4263]
As a workaround, we can use {{appendZoneOrOffsetId()}} rather than
{{appendZoneText().}}
This ensures the tests are passing for Java 8+ and based on my testing, I
didn’t see any regression of the change.
Sample repro code - jdoodle.com/ia/D5e
was:
Some of the unit tests related to _DateTimeFormatter_ were failing in Java
versions greater than 8 while working in Java 8.
Example of a failing Unit Test :
_org.apache.hadoop.hive.common.type.TestTimestampTZ#testComparision_
{code:java}
java.time.format.DateTimeParseException: Text '2017-04-14 18:00:00 UTC+08:00'
could not be parsed, unparsed text found at index 23 at
java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
at
java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
at
org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:76)
at
org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:64)
at
org.apache.hadoop.hive.common.type.TestTimestampTZ.testComparison(TestTimestampTZ.java:44)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method){code}
The {{appendZoneText(TextStyle)}} method of
[DateTimeFormatteBuilder|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java#L82]
is not able to parse the {{+}} symbol in {{2017-04-14 18:00:00 UTC+08:00}}
when running [the
test|https://github.com/apache/hive/blob/master/common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java#L37]
in Java 11 , while it is working fine with Java 8.
According to the
[doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendZoneText(java.time.format.TextStyle)]
, {{appendZoneText(TextStyle)}} should be able to parse {{either the textual
zone name, the zone ID or the offset}} and UTC+08:00 should come under offset
as per the same
[doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendOffset(java.lang.String,%20java.lang.String)].
It seems, however, that "UTC" was explicitly removed [when parsed as ZoneText
for higher Java
versions|https://github.com/openjdk/jdk/commit/5c3a01591c5c945926636fdc9f164d60b5b4f29e?diff=unified#diff-5fcf976db1c06e8f44a8671356d7e34fdfbf5b057baa852e7c3e015c8797c889R4263]
As a workaround, we can use {{appendZoneOrOffsetId()}} rather than
{{appendZoneText().}}
This ensures the tests are passing for Java 8+ and based on my testing, I
didn’t see any regression of the change.
> DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions
[jira] [Commented] (HIVE-26793) Create a new configuration to override "no compaction" for tables
[
https://issues.apache.org/jira/browse/HIVE-26793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682395#comment-17682395
]
Kokila N commented on HIVE-26793:
-
Okay., I will update with the changes.
> Create a new configuration to override "no compaction" for tables
> -
>
> Key: HIVE-26793
> URL: https://issues.apache.org/jira/browse/HIVE-26793
> Project: Hive
> Issue Type: Improvement
>Reporter: Kokila N
>Assignee: Kokila N
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Currently a simple user can create a table with
> {color:#6a8759}no_auto_compaction=true{color} table property and create an
> aborted write transaction writing to this table. This way a malicious user
> can prevent cleaning up data for the aborted transaction, creating
> performance degradation.
> This configuration should be allowed to overridden on a database level:
> adding {color:#6a8759}no_auto_compaction=false{color} should override the
> table level setting forcing the initiator to schedule compaction for all
> tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26960) Optimized bootstrap does not drop newly added tables at source.
[ https://issues.apache.org/jira/browse/HIVE-26960?focusedWorklogId=842426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842426 ] ASF GitHub Bot logged work on HIVE-26960: - Author: ASF GitHub Bot Created on: 31/Jan/23 05:38 Start Date: 31/Jan/23 05:38 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3995: URL: https://github.com/apache/hive/pull/3995#issuecomment-1409792967 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3995) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3995=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3995=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3995=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=CODE_SMELL) [3 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3995=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3995=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842426) Time Spent: 40m (was: 0.5h) > Optimized bootstrap does not drop newly added tables at source. > --- > > Key: HIVE-26960 > URL: https://issues.apache.org/jira/browse/HIVE-26960 > Project: Hive > Issue Type: Bug >Reporter: Rakshith C >Assignee: Rakshith C >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Scenario: > Replication is setup from DR to PROD after failover from PROD to DR and no > existing tables are modified at PROD but a new table is added at PROD. > Observations: > * _bootstrap directory won't be created during second cycle of optimized > bootstrap because existing tables were not modified. > * Based on this, it will not initialize list of tables to drop at PROD. > * This leads to the new table created at PROD not being dropped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27004) DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
[
https://issues.apache.org/jira/browse/HIVE-27004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anmol Sundaram updated HIVE-27004:
--
Attachment: HIVE-27004.patch
> DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
> ---
>
> Key: HIVE-27004
> URL: https://issues.apache.org/jira/browse/HIVE-27004
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: Java version used
> {code:java}
> openjdk version "11.0.16.1" 2022-08-12 LTS OpenJDK Runtime Environment
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS) OpenJDK 64-Bit Server VM
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS, mixed mode{code}
>
>Reporter: Anmol Sundaram
>Assignee: Anmol Sundaram
>Priority: Minor
> Attachments: HIVE-27004.patch
>
>
> Some of the unit tests related to _DateTimeFormatter_ were failing in Java
> versions greater than 8 while working in Java 8.
> Example of a failing Unit Test :
> _org.apache.hadoop.hive.common.type.TestTimestampTZ#testComparision_
>
> {code:java}
> java.time.format.DateTimeParseException: Text '2017-04-14 18:00:00 UTC+08:00'
> could not be parsed, unparsed text found at index 23 at
> java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:76)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:64)
> at
> org.apache.hadoop.hive.common.type.TestTimestampTZ.testComparison(TestTimestampTZ.java:44)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method){code}
>
> The {{appendZoneText(TextStyle)}} method of
> [DateTimeFormatteBuilder|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java#L82]
> is not able to parse the {{+}} symbol in {{2017-04-14 18:00:00 UTC+08:00}}
> when running [the
> test|https://github.com/apache/hive/blob/master/common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java#L37]
> in Java 11 , while it is working fine with Java 8.
> According to the
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendZoneText(java.time.format.TextStyle)]
> , {{appendZoneText(TextStyle)}} should be able to parse {{either the textual
> zone name, the zone ID or the offset}} and UTC+08:00 should come under offset
> as per the same
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendOffset(java.lang.String,%20java.lang.String)].
> It seems, however, that "UTC" was explicitly removed [when parsed as ZoneText
> for higher Java
> versions|https://github.com/openjdk/jdk/commit/5c3a01591c5c945926636fdc9f164d60b5b4f29e?diff=unified#diff-5fcf976db1c06e8f44a8671356d7e34fdfbf5b057baa852e7c3e015c8797c889R4263]
> As a workaround, we can use {{appendZoneOrOffsetId()}} rather than
> {{appendZoneText().}}
> This ensures the tests are passing for Java 8+ and based on my testing, I
> didn’t see any regression of the change.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27004) DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
[
https://issues.apache.org/jira/browse/HIVE-27004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anmol Sundaram updated HIVE-27004:
--
Environment:
Java version used
{code:java}
openjdk version "11.0.16.1" 2022-08-12 LTS OpenJDK Runtime Environment
Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS) OpenJDK 64-Bit Server VM
Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS, mixed mode{code}
was:
Java version used
{{}}
{code:java}
openjdk version "11.0.16.1" 2022-08-12 LTS OpenJDK Runtime Environment
Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS) OpenJDK 64-Bit Server VM
Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS, mixed mode{code}
{{}}
> DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
> ---
>
> Key: HIVE-27004
> URL: https://issues.apache.org/jira/browse/HIVE-27004
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: Java version used
> {code:java}
> openjdk version "11.0.16.1" 2022-08-12 LTS OpenJDK Runtime Environment
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS) OpenJDK 64-Bit Server VM
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS, mixed mode{code}
>
>Reporter: Anmol Sundaram
>Assignee: Anmol Sundaram
>Priority: Minor
>
> Some of the unit tests related to _DateTimeFormatter_ were failing in Java
> versions greater than 8 while working in Java 8.
> Example of a failing Unit Test :
> _org.apache.hadoop.hive.common.type.TestTimestampTZ#testComparision_
>
> {code:java}
> java.time.format.DateTimeParseException: Text '2017-04-14 18:00:00 UTC+08:00'
> could not be parsed, unparsed text found at index 23 at
> java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:76)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:64)
> at
> org.apache.hadoop.hive.common.type.TestTimestampTZ.testComparison(TestTimestampTZ.java:44)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method){code}
>
> The {{appendZoneText(TextStyle)}} method of
> [DateTimeFormatteBuilder|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java#L82]
> is not able to parse the {{+}} symbol in {{2017-04-14 18:00:00 UTC+08:00}}
> when running [the
> test|https://github.com/apache/hive/blob/master/common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java#L37]
> in Java 11 , while it is working fine with Java 8.
> According to the
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendZoneText(java.time.format.TextStyle)]
> , {{appendZoneText(TextStyle)}} should be able to parse {{either the textual
> zone name, the zone ID or the offset}} and UTC+08:00 should come under offset
> as per the same
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendOffset(java.lang.String,%20java.lang.String)].
> It seems, however, that "UTC" was explicitly removed [when parsed as ZoneText
> for higher Java
> versions|https://github.com/openjdk/jdk/commit/5c3a01591c5c945926636fdc9f164d60b5b4f29e?diff=unified#diff-5fcf976db1c06e8f44a8671356d7e34fdfbf5b057baa852e7c3e015c8797c889R4263]
> As a workaround, we can use {{appendZoneOrOffsetId()}} rather than
> {{appendZoneText().}}
> This ensures the tests are passing for Java 8+ and based on my testing, I
> didn’t see any regression of the change.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27004) DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
[
https://issues.apache.org/jira/browse/HIVE-27004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anmol Sundaram reassigned HIVE-27004:
-
> DateTimeFormatterBuilder cannot parse 'UTC+' in Java versions higher than 8
> ---
>
> Key: HIVE-27004
> URL: https://issues.apache.org/jira/browse/HIVE-27004
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Environment: Java version used
> {{}}
> {code:java}
> openjdk version "11.0.16.1" 2022-08-12 LTS OpenJDK Runtime Environment
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS) OpenJDK 64-Bit Server VM
> Corretto-11.0.16.9.1 (build 11.0.16.1+9-LTS, mixed mode{code}
> {{}}
>Reporter: Anmol Sundaram
>Assignee: Anmol Sundaram
>Priority: Minor
>
> Some of the unit tests related to _DateTimeFormatter_ were failing in Java
> versions greater than 8 while working in Java 8.
> Example of a failing Unit Test :
> _org.apache.hadoop.hive.common.type.TestTimestampTZ#testComparision_
>
> {code:java}
> java.time.format.DateTimeParseException: Text '2017-04-14 18:00:00 UTC+08:00'
> could not be parsed, unparsed text found at index 23 at
> java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2049)
> at
> java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1874)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:76)
> at
> org.apache.hadoop.hive.common.type.TimestampTZUtil.parse(TimestampTZUtil.java:64)
> at
> org.apache.hadoop.hive.common.type.TestTimestampTZ.testComparison(TestTimestampTZ.java:44)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method){code}
>
> The {{appendZoneText(TextStyle)}} method of
> [DateTimeFormatteBuilder|https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/type/TimestampTZUtil.java#L82]
> is not able to parse the {{+}} symbol in {{2017-04-14 18:00:00 UTC+08:00}}
> when running [the
> test|https://github.com/apache/hive/blob/master/common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java#L37]
> in Java 11 , while it is working fine with Java 8.
> According to the
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendZoneText(java.time.format.TextStyle)]
> , {{appendZoneText(TextStyle)}} should be able to parse {{either the textual
> zone name, the zone ID or the offset}} and UTC+08:00 should come under offset
> as per the same
> [doc|https://developer.android.com/reference/java/time/format/DateTimeFormatterBuilder#appendOffset(java.lang.String,%20java.lang.String)].
> It seems, however, that "UTC" was explicitly removed [when parsed as ZoneText
> for higher Java
> versions|https://github.com/openjdk/jdk/commit/5c3a01591c5c945926636fdc9f164d60b5b4f29e?diff=unified#diff-5fcf976db1c06e8f44a8671356d7e34fdfbf5b057baa852e7c3e015c8797c889R4263]
> As a workaround, we can use {{appendZoneOrOffsetId()}} rather than
> {{appendZoneText().}}
> This ensures the tests are passing for Java 8+ and based on my testing, I
> didn’t see any regression of the change.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26606) Expose failover states in replication metrics
[ https://issues.apache.org/jira/browse/HIVE-26606?focusedWorklogId=842419=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842419 ] ASF GitHub Bot logged work on HIVE-26606: - Author: ASF GitHub Bot Created on: 31/Jan/23 04:53 Start Date: 31/Jan/23 04:53 Worklog Time Spent: 10m Work Description: pudidic commented on PR #3990: URL: https://github.com/apache/hive/pull/3990#issuecomment-1409752367 +1. LGTM. Issue Time Tracking --- Worklog Id: (was: 842419) Time Spent: 1h 50m (was: 1h 40m) > Expose failover states in replication metrics > - > > Key: HIVE-26606 > URL: https://issues.apache.org/jira/browse/HIVE-26606 > Project: Hive > Issue Type: Improvement >Reporter: Teddy Choi >Assignee: Harshal Patel >Priority: Major > Labels: pull-request-available > Attachments: > HIVE-26606__Expose_failover_states_in_replication_metrics1.patch > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Expose the state of failover in replication metrics, -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26606) Expose failover states in replication metrics
[ https://issues.apache.org/jira/browse/HIVE-26606?focusedWorklogId=842420=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842420 ] ASF GitHub Bot logged work on HIVE-26606: - Author: ASF GitHub Bot Created on: 31/Jan/23 04:53 Start Date: 31/Jan/23 04:53 Worklog Time Spent: 10m Work Description: pudidic merged PR #3990: URL: https://github.com/apache/hive/pull/3990 Issue Time Tracking --- Worklog Id: (was: 842420) Time Spent: 2h (was: 1h 50m) > Expose failover states in replication metrics > - > > Key: HIVE-26606 > URL: https://issues.apache.org/jira/browse/HIVE-26606 > Project: Hive > Issue Type: Improvement >Reporter: Teddy Choi >Assignee: Harshal Patel >Priority: Major > Labels: pull-request-available > Attachments: > HIVE-26606__Expose_failover_states_in_replication_metrics1.patch > > Time Spent: 2h > Remaining Estimate: 0h > > Expose the state of failover in replication metrics, -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842413=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842413
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 31/Jan/23 03:50
Start Date: 31/Jan/23 03:50
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3905:
URL: https://github.com/apache/hive/pull/3905#issuecomment-1409712790
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3905)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3905=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3905=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3905=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=CODE_SMELL)
[5 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3905=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3905=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 842413)
Time Spent: 7h 20m (was: 7h 10m)
> Explore moving to directsql for ObjectStore::addPartitions
> --
>
> Key: HIVE-26035
> URL: https://issues.apache.org/jira/browse/HIVE-26035
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Venugopal Reddy K
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> Currently {{addPartitions}} uses datanuclues and is super slow for large
> number of partitions. It will be good to move to direct sql. Lots of repeated
> SQLs can be avoided as well (e.g SDS, SERDE, TABLE_PARAMS)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26986) A DAG created by OperatorGraph is not equal to the Tez DAG.
[ https://issues.apache.org/jira/browse/HIVE-26986?focusedWorklogId=842409=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842409 ] ASF GitHub Bot logged work on HIVE-26986: - Author: ASF GitHub Bot Created on: 31/Jan/23 03:28 Start Date: 31/Jan/23 03:28 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3998: URL: https://github.com/apache/hive/pull/3998#issuecomment-1409697045 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3998) [](https://sonarcloud.io/project/issues?id=apache_hive=3998=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3998=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3998=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3998=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3998=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3998=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3998=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3998=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3998=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3998=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3998=false=CODE_SMELL) [14 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3998=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3998=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3998=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842409) Time Spent: 20m (was: 10m) > A DAG created by OperatorGraph is not equal to the Tez DAG. > --- > > Key: HIVE-26986 > URL: https://issues.apache.org/jira/browse/HIVE-26986 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-2 >Reporter: Seonggon Namgung >Assignee: Seonggon Namgung >Priority: Major > Labels: pull-request-available > Attachments: Query71 OperatorGraph.png, Query71 TezDAG.png > > Time Spent: 20m > Remaining Estimate: 0h > > A DAG created by OperatorGraph is not equal to the corresponding DAG that is > submitted to Tez. > Because of this problem, ParallelEdgeFixer reports a pair of normal edges to > a parallel edge. > We observe this problem by comparing OperatorGraph and Tez DAG when running > TPC-DS query 71 on 1TB ORC format managed table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26986) A DAG created by OperatorGraph is not equal to the Tez DAG.
[ https://issues.apache.org/jira/browse/HIVE-26986?focusedWorklogId=842407=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842407 ] ASF GitHub Bot logged work on HIVE-26986: - Author: ASF GitHub Bot Created on: 31/Jan/23 02:43 Start Date: 31/Jan/23 02:43 Worklog Time Spent: 10m Work Description: ngsg opened a new pull request, #3998: URL: https://github.com/apache/hive/pull/3998 ### What changes were proposed in this pull request? I rewrite OperatorGraph to make sure that OperatorGraph represents actual Tez DAG. In new OperatorGraph, each cluster has a unique root operator and one Operator can belong to more than one cluster. ### Why are the changes needed? Tez DAG and current OperatorGraph group operators differently when a query plan contains union operators. More precisely, OperatorGraph groups all root operators of a union operator into the same cluster, but Tez DAG creates a BaseWork for each of the root operators and clones descendants of the union operator if needed. This difference makes false-positive errors when detecting parallel edges, which leads to insertion of unnecessary concentrator RS by ParallelEdgeFixer. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? I run the below maven tests and add a unit test for OperatorGraph. ``` mvn test -Dtest=TestSharedWorkOptimizer cd itests; mvn test -Dtest=TestCliDriver ``` Issue Time Tracking --- Worklog Id: (was: 842407) Remaining Estimate: 0h Time Spent: 10m > A DAG created by OperatorGraph is not equal to the Tez DAG. > --- > > Key: HIVE-26986 > URL: https://issues.apache.org/jira/browse/HIVE-26986 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-2 >Reporter: Seonggon Namgung >Assignee: Seonggon Namgung >Priority: Major > Attachments: Query71 OperatorGraph.png, Query71 TezDAG.png > > Time Spent: 10m > Remaining Estimate: 0h > > A DAG created by OperatorGraph is not equal to the corresponding DAG that is > submitted to Tez. > Because of this problem, ParallelEdgeFixer reports a pair of normal edges to > a parallel edge. > We observe this problem by comparing OperatorGraph and Tez DAG when running > TPC-DS query 71 on 1TB ORC format managed table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26986) A DAG created by OperatorGraph is not equal to the Tez DAG.
[ https://issues.apache.org/jira/browse/HIVE-26986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26986: -- Labels: pull-request-available (was: ) > A DAG created by OperatorGraph is not equal to the Tez DAG. > --- > > Key: HIVE-26986 > URL: https://issues.apache.org/jira/browse/HIVE-26986 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-2 >Reporter: Seonggon Namgung >Assignee: Seonggon Namgung >Priority: Major > Labels: pull-request-available > Attachments: Query71 OperatorGraph.png, Query71 TezDAG.png > > Time Spent: 10m > Remaining Estimate: 0h > > A DAG created by OperatorGraph is not equal to the corresponding DAG that is > submitted to Tez. > Because of this problem, ParallelEdgeFixer reports a pair of normal edges to > a parallel edge. > We observe this problem by comparing OperatorGraph and Tez DAG when running > TPC-DS query 71 on 1TB ORC format managed table. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26455) Remove PowerMockito from hive-exec
[ https://issues.apache.org/jira/browse/HIVE-26455?focusedWorklogId=842402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842402 ] ASF GitHub Bot logged work on HIVE-26455: - Author: ASF GitHub Bot Created on: 31/Jan/23 00:22 Start Date: 31/Jan/23 00:22 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #3798: HIVE-26455: Remove powermock from hive-exec URL: https://github.com/apache/hive/pull/3798 Issue Time Tracking --- Worklog Id: (was: 842402) Time Spent: 1h (was: 50m) > Remove PowerMockito from hive-exec > -- > > Key: HIVE-26455 > URL: https://issues.apache.org/jira/browse/HIVE-26455 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Zsolt Miskolczi >Assignee: Zsolt Miskolczi >Priority: Minor > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > PowerMockito is a mockito extension that introduces some painful points. > The main intention behind that is to be able to do static mocking. Since its > release, mockito-inline has been released, as a part of the mockito-core. > It doesn't require vintage test runner to be able to run and it can mock > objects with their own thread. > The goal is to stop using PowerMockito and use mockito-inline instead. > > The affected packages are: > * org.apache.hadoop.hive.ql.exec.repl > * org.apache.hadoop.hive.ql.exec.repl.bootstrap.load > * org.apache.hadoop.hive.ql.exec.repl.ranger; > * org.apache.hadoop.hive.ql.exec.util > * org.apache.hadoop.hive.ql.parse.repl > * org.apache.hadoop.hive.ql.parse.repl.load.message > * org.apache.hadoop.hive.ql.parse.repl.metric > * org.apache.hadoop.hive.ql.txn.compactor > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26774) Implement array_slice UDF to get the subset of elements from an array (subarray)
[
https://issues.apache.org/jira/browse/HIVE-26774?focusedWorklogId=842389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842389
]
ASF GitHub Bot logged work on HIVE-26774:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 22:17
Start Date: 30/Jan/23 22:17
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera merged PR #3893:
URL: https://github.com/apache/hive/pull/3893
Issue Time Tracking
---
Worklog Id: (was: 842389)
Time Spent: 1h 40m (was: 1.5h)
> Implement array_slice UDF to get the subset of elements from an array
> (subarray)
>
>
> Key: HIVE-26774
> URL: https://issues.apache.org/jira/browse/HIVE-26774
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> *array_slice(array, start, length)* - Returns the subset or range of elements
> from an array (subarray).
> Example:
>
> {noformat}
> > SELECT array_slice(array(1, 2, 3,4), 2,2) FROM src LIMIT 1;
> 3,4{noformat}
> Returns empty list if start/length are out of range of the array
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842387=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842387
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 22:03
Start Date: 30/Jan/23 22:03
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3905:
URL: https://github.com/apache/hive/pull/3905#issuecomment-1409418956
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3905)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3905=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3905=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3905=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=CODE_SMELL)
[5 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3905=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3905=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 842387)
Time Spent: 7h 10m (was: 7h)
> Explore moving to directsql for ObjectStore::addPartitions
> --
>
> Key: HIVE-26035
> URL: https://issues.apache.org/jira/browse/HIVE-26035
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Venugopal Reddy K
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Currently {{addPartitions}} uses datanuclues and is super slow for large
> number of partitions. It will be good to move to direct sql. Lots of repeated
> SQLs can be avoided as well (e.g SDS, SERDE, TABLE_PARAMS)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26757) Add sfs+ofs support
[
https://issues.apache.org/jira/browse/HIVE-26757?focusedWorklogId=842386=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842386
]
ASF GitHub Bot logged work on HIVE-26757:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 21:40
Start Date: 30/Jan/23 21:40
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3779:
URL: https://github.com/apache/hive/pull/3779#issuecomment-1409386094
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3779)
[](https://sonarcloud.io/project/issues?id=apache_hive=3779=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3779=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3779=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3779=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3779=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3779=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3779=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3779=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3779=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3779=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3779=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3779=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3779=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3779=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 842386)
Time Spent: 1h 50m (was: 1h 40m)
> Add sfs+ofs support
> ---
>
> Key: HIVE-26757
> URL: https://issues.apache.org/jira/browse/HIVE-26757
> Project: Hive
> Issue Type: Improvement
>Reporter: Michael Smith
>Assignee: Michael Smith
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> [https://github.com/apache/hive/blob/ebb1e2fa9914bcccecad261d53338933b699ccb1/ql/src/java/org/apache/hadoop/hive/ql/io/SingleFileSystem.java#L80]
> shows SFS support for Ozone's o3fs protocol, but not the newer ofs protocol.
> Please add support for {{{}sfs+ofs{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26996) typos in hive-exec
[ https://issues.apache.org/jira/browse/HIVE-26996?focusedWorklogId=842384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842384 ] ASF GitHub Bot logged work on HIVE-26996: - Author: ASF GitHub Bot Created on: 30/Jan/23 21:30 Start Date: 30/Jan/23 21:30 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3994: URL: https://github.com/apache/hive/pull/3994#issuecomment-1409370015 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3994) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3994=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3994=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3994=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3994=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3994=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842384) Time Spent: 0.5h (was: 20m) > typos in hive-exec > -- > > Key: HIVE-26996 > URL: https://issues.apache.org/jira/browse/HIVE-26996 > Project: Hive > Issue Type: Improvement > Components: Query Planning, Query Processor >Affects Versions: All Versions >Reporter: Michal Lorek >Assignee: Michal Lorek >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > typos and grammar errors in hive-exec module -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26980) Creation of materialized view stored by Iceberg fails if source table has tinyint column
[
https://issues.apache.org/jira/browse/HIVE-26980?focusedWorklogId=842377=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842377
]
ASF GitHub Bot logged work on HIVE-26980:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 20:44
Start Date: 30/Jan/23 20:44
Worklog Time Spent: 10m
Work Description: InvisibleProgrammer commented on PR #3993:
URL: https://github.com/apache/hive/pull/3993#issuecomment-1409307463
Hi, @kasakrisz !
Thank you for fixing that issue. I have faced with the exact same in
https://issues.apache.org/jira/browse/HIVE-26977?
Could you please add that qtest as well to validate your fix fixes the other
issue as well. I already tested it on my computer and it worked.
```
set iceberg.mr.schema.auto.conversion=true;
set hive.vectorized.execution.enabled=true;
set hive.explain.user=false;
drop table if exists source_table;
create external table source_table
(
id char(16)
);
insert into source_table values ('ID_1');
insert into source_table values ('ID_2');
drop table if exists target_table;
explain
create table target_table
stored by iceberg
stored as orc
as select * from source_table;
create table target_table
stored by iceberg
stored as orc
as select * from source_table;
select count(*) from target_table;
```
Thank you,
Zsolt
Issue Time Tracking
---
Worklog Id: (was: 842377)
Time Spent: 1h 10m (was: 1h)
> Creation of materialized view stored by Iceberg fails if source table has
> tinyint column
>
>
> Key: HIVE-26980
> URL: https://issues.apache.org/jira/browse/HIVE-26980
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> MV creation using iceberg storage format for Hive table containing tinyint
> column fails with error "java.lang.ClassCastException: class java.lang.Byte
> cannot be cast to class java.lang.Integer"
> Logs
> {code:java}
> 2023-01-24T21:36:22,153 INFO [pool-2-thread-1] jdbc.TestDriver: Beginning
> Test at 2023-01-24 21:36:22,153
> 2023-01-24T21:36:22,154 INFO [pool-2-thread-1] jdbc.TestDriver: BEGIN MAIN
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Running
> SessionGroup{name='SG_TNMECIDQ31', initialDelay=0, repeats=1, repeatDelay=0}
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Connecting
> as user 'hrt_qa'
> 2023-01-24T21:36:22,409 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.cbo.enable=true
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.materializedview.rewriting
> 2023-01-24T21:36:22,946 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> iceberg.mr.schema.auto.conversion=true
> 2023-01-24T21:36:23,183 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:23,183 INFO [pool-38-thread-1] jdbc.TestDriver: Query: drop
> materialized view if exists mv_project
> 2023-01-24T21:36:23,243 INFO [Thread-401] jdbc.TestDriver: INFO : Compiling
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3):
> drop materialized view if exists mv_project
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Semantic
> Analysis Completed (retrial = false)
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Created
> Hive schema: Schema(fieldSchemas:null, properties:null)
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Completed
> compiling
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3);
> Time taken: 0.03 seconds
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Executing
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3):
> drop materialized view if exists mv_project
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Starting
> task [Stage-0:DDL] in serial mode
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : Completed
> executing
> command(queryId=hive_20230124213623_e80df624-1ca6-4487-9f4b-616f756f91b3);
> Time taken: 0.008 seconds
> 2023-01-24T21:36:23,302 INFO [Thread-401] jdbc.TestDriver: INFO : OK
> 2023-01-24T21:36:23,423 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:23,423 INFO [pool-38-thread-1] jdbc.TestDriver: Query:
> create materialized
[jira] [Work logged] (HIVE-27000) Improve the modularity of the *ColumnStatsMerger classes
[ https://issues.apache.org/jira/browse/HIVE-27000?focusedWorklogId=842373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842373 ] ASF GitHub Bot logged work on HIVE-27000: - Author: ASF GitHub Bot Created on: 30/Jan/23 20:10 Start Date: 30/Jan/23 20:10 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3997: URL: https://github.com/apache/hive/pull/3997#issuecomment-1409264433 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3997) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3997=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3997=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3997=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3997=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3997=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3997=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842373) Time Spent: 20m (was: 10m) > Improve the modularity of the *ColumnStatsMerger classes > > > Key: HIVE-27000 > URL: https://issues.apache.org/jira/browse/HIVE-27000 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > *ColumnStatsMerger classes contain a lot of duplicate code which is not > specific to the data type, and that could therefore be lifted to a common > parent class. > This phenomenon is bound to become even worse if we keep enriching further > our supported set of statistics as we did in the context of HIVE-26221. > The current ticket aims at improving the modularity and code reuse of the > *ColumnStatsMerger classes, while improving unit-test coverage to cover all > classes and support more use-cases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26998) Can not find correct root task on compute column statistics for views
[ https://issues.apache.org/jira/browse/HIVE-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682253#comment-17682253 ] Sai Hemanth Gantasala commented on HIVE-26998: -- cc [~krisztiankasa] > Can not find correct root task on compute column statistics for views > - > > Key: HIVE-26998 > URL: https://issues.apache.org/jira/browse/HIVE-26998 > Project: Hive > Issue Type: Bug >Reporter: chiranjeevi >Priority: Minor > > When you run ANALYZE TABLE view_name COMPUTE STATISTICS FOR COLUMNS; > Server should notify > *Expected Result :* > ANALYZE is not supported for views (state=42000,code=10091) like it does for > table stat > *Actual result:* > Error: Error while compiling statement: FAILED: SemanticException Can not > find correct root task -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26908) Provide configs to selectively enable/disable Initiator/Cleaner threads
[
https://issues.apache.org/jira/browse/HIVE-26908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko resolved HIVE-26908.
---
Fix Version/s: 4.0.0
Resolution: Fixed
> Provide configs to selectively enable/disable Initiator/Cleaner threads
> ---
>
> Key: HIVE-26908
> URL: https://issues.apache.org/jira/browse/HIVE-26908
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> In the current implementation, both Initiator and Cleaner are either enabled
> or disabled using the same config
> {noformat}
> hive.compactor.initiator.on{noformat}
> So there is no way to selectively disable initiator and enable cleaner or
> vice versa.
> Introducing another config to handle Cleaner thread alone like
> hive.compactor.cleaner.on
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26908) Provide configs to selectively enable/disable Initiator/Cleaner threads
[
https://issues.apache.org/jira/browse/HIVE-26908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682244#comment-17682244
]
Denys Kuzmenko commented on HIVE-26908:
---
Merget to master.
[~tarak271] thanks for the patch and [~akshatm], [~lvegh] for the review!
> Provide configs to selectively enable/disable Initiator/Cleaner threads
> ---
>
> Key: HIVE-26908
> URL: https://issues.apache.org/jira/browse/HIVE-26908
> Project: Hive
> Issue Type: New Feature
> Components: Standalone Metastore
>Reporter: Taraka Rama Rao Lethavadla
>Assignee: Taraka Rama Rao Lethavadla
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> In the current implementation, both Initiator and Cleaner are either enabled
> or disabled using the same config
> {noformat}
> hive.compactor.initiator.on{noformat}
> So there is no way to selectively disable initiator and enable cleaner or
> vice versa.
> Introducing another config to handle Cleaner thread alone like
> hive.compactor.cleaner.on
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27000) Improve the modularity of the *ColumnStatsMerger classes
[ https://issues.apache.org/jira/browse/HIVE-27000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27000: -- Labels: pull-request-available (was: ) > Improve the modularity of the *ColumnStatsMerger classes > > > Key: HIVE-27000 > URL: https://issues.apache.org/jira/browse/HIVE-27000 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > *ColumnStatsMerger classes contain a lot of duplicate code which is not > specific to the data type, and that could therefore be lifted to a common > parent class. > This phenomenon is bound to become even worse if we keep enriching further > our supported set of statistics as we did in the context of HIVE-26221. > The current ticket aims at improving the modularity and code reuse of the > *ColumnStatsMerger classes, while improving unit-test coverage to cover all > classes and support more use-cases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-27000) Improve the modularity of the *ColumnStatsMerger classes
[ https://issues.apache.org/jira/browse/HIVE-27000?focusedWorklogId=842367=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842367 ] ASF GitHub Bot logged work on HIVE-27000: - Author: ASF GitHub Bot Created on: 30/Jan/23 18:47 Start Date: 30/Jan/23 18:47 Worklog Time Spent: 10m Work Description: asolimando opened a new pull request, #3997: URL: https://github.com/apache/hive/pull/3997 ### What changes were proposed in this pull request? Improving the modularity of the *ColumnStatsMerger classes by introducing a common parent class and moving there all the shared code that is not depending on the data type. ### Why are the changes needed? Those classes host some complex merging logic that is copy-pasted into several classes, a problem that is bound to worsen as we add new (and more complex) statistics. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing and introduced unit-tests relevant for the change: ```mvn test -Dtest.groups=org.apache.hadoop.hive.metastore.annotation.MetastoreUnitTest -Dtest='*ColumnStats*Test.java' -pl standalone-metastore/metastore-server``` Issue Time Tracking --- Worklog Id: (was: 842367) Remaining Estimate: 0h Time Spent: 10m > Improve the modularity of the *ColumnStatsMerger classes > > > Key: HIVE-27000 > URL: https://issues.apache.org/jira/browse/HIVE-27000 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > *ColumnStatsMerger classes contain a lot of duplicate code which is not > specific to the data type, and that could therefore be lifted to a common > parent class. > This phenomenon is bound to become even worse if we keep enriching further > our supported set of statistics as we did in the context of HIVE-26221. > The current ticket aims at improving the modularity and code reuse of the > *ColumnStatsMerger classes, while improving unit-test coverage to cover all > classes and support more use-cases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26898) Split Notification logging so that busy clusters can have better performance
[ https://issues.apache.org/jira/browse/HIVE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682238#comment-17682238 ] Riju Trivedi commented on HIVE-26898: - [~ayushtkn] Thanks for pointing this out. This comment was meant for another internal jira, mistakenly posted here. > Split Notification logging so that busy clusters can have better performance > > > Key: HIVE-26898 > URL: https://issues.apache.org/jira/browse/HIVE-26898 > Project: Hive > Issue Type: New Feature >Reporter: Taraka Rama Rao Lethavadla >Priority: Major > > For DDL & DML events are logged into notifications log table and those get > cleaned as soon as ttl got expired. > In most of the busy clusters, the notification log is growing even though > cleaner is running and kept on cleaning the events. It means the rate of Hive > db operations are very high compared to rate at which cleaning is happening. > So any query on this table is becoming bottle neck at backend DB causing slow > response > The proposal is to split the notification log table in to multiple tables > like > notification_log_dml - for all DML queries > notification_log_insert - for all insert queries > .. > etc. > > So that load on that single table gets reduced improving the performance of > the backend db as well as Hive -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26898) Split Notification logging so that busy clusters can have better performance
[
https://issues.apache.org/jira/browse/HIVE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682236#comment-17682236
]
Ayush Saxena commented on HIVE-26898:
-
{quote}OOM issues In recent versions 717, SP1 and SP2?
{quote}
There is no version like this, these are the only versions that we have here:
[https://hive.apache.org/general/downloads/]
> Split Notification logging so that busy clusters can have better performance
>
>
> Key: HIVE-26898
> URL: https://issues.apache.org/jira/browse/HIVE-26898
> Project: Hive
> Issue Type: New Feature
>Reporter: Taraka Rama Rao Lethavadla
>Priority: Major
>
> For DDL & DML events are logged into notifications log table and those get
> cleaned as soon as ttl got expired.
> In most of the busy clusters, the notification log is growing even though
> cleaner is running and kept on cleaning the events. It means the rate of Hive
> db operations are very high compared to rate at which cleaning is happening.
> So any query on this table is becoming bottle neck at backend DB causing slow
> response
> The proposal is to split the notification log table in to multiple tables
> like
> notification_log_dml - for all DML queries
> notification_log_insert - for all insert queries
> ..
> etc.
>
> So that load on that single table gets reduced improving the performance of
> the backend db as well as Hive
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-27001) Backport HIVE-26633 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-27001: -- Target Version/s: 3.2.0 Assignee: Vihang Karajgaonkar Priority: Blocker (was: Major) > Backport HIVE-26633 to branch-3 > --- > > Key: HIVE-27001 > URL: https://issues.apache.org/jira/browse/HIVE-27001 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Blocker > > HIVE-26633 fixes the maximum response size in metastore http mode. We should > backport this to branch-3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-27000) Improve the modularity of the *ColumnStatsMerger classes
[ https://issues.apache.org/jira/browse/HIVE-27000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-27000 started by Alessandro Solimando. --- > Improve the modularity of the *ColumnStatsMerger classes > > > Key: HIVE-27000 > URL: https://issues.apache.org/jira/browse/HIVE-27000 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > > *ColumnStatsMerger classes contain a lot of duplicate code which is not > specific to the data type, and that could therefore be lifted to a common > parent class. > This phenomenon is bound to become even worse if we keep enriching further > our supported set of statistics as we did in the context of HIVE-26221. > The current ticket aims at improving the modularity and code reuse of the > *ColumnStatsMerger classes, while improving unit-test coverage to cover all > classes and support more use-cases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-27000) Improve the modularity of the *ColumnStatsMerger classes
[ https://issues.apache.org/jira/browse/HIVE-27000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alessandro Solimando reassigned HIVE-27000: --- > Improve the modularity of the *ColumnStatsMerger classes > > > Key: HIVE-27000 > URL: https://issues.apache.org/jira/browse/HIVE-27000 > Project: Hive > Issue Type: Improvement > Components: Statistics >Affects Versions: 4.0.0-alpha-2 >Reporter: Alessandro Solimando >Assignee: Alessandro Solimando >Priority: Major > > *ColumnStatsMerger classes contain a lot of duplicate code which is not > specific to the data type, and that could therefore be lifted to a common > parent class. > This phenomenon is bound to become even worse if we keep enriching further > our supported set of statistics as we did in the context of HIVE-26221. > The current ticket aims at improving the modularity and code reuse of the > *ColumnStatsMerger classes, while improving unit-test coverage to cover all > classes and support more use-cases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26980) Creation of materialized view stored by Iceberg fails if source table has tinyint column
[
https://issues.apache.org/jira/browse/HIVE-26980?focusedWorklogId=842358=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842358
]
ASF GitHub Bot logged work on HIVE-26980:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 18:07
Start Date: 30/Jan/23 18:07
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3993:
URL: https://github.com/apache/hive/pull/3993#issuecomment-1409083861
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3993)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3993=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3993=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3993=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3993=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3993=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 842358)
Time Spent: 1h (was: 50m)
> Creation of materialized view stored by Iceberg fails if source table has
> tinyint column
>
>
> Key: HIVE-26980
> URL: https://issues.apache.org/jira/browse/HIVE-26980
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> MV creation using iceberg storage format for Hive table containing tinyint
> column fails with error "java.lang.ClassCastException: class java.lang.Byte
> cannot be cast to class java.lang.Integer"
> Logs
> {code:java}
> 2023-01-24T21:36:22,153 INFO [pool-2-thread-1] jdbc.TestDriver: Beginning
> Test at 2023-01-24 21:36:22,153
> 2023-01-24T21:36:22,154 INFO [pool-2-thread-1] jdbc.TestDriver: BEGIN MAIN
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Running
> SessionGroup{name='SG_TNMECIDQ31', initialDelay=0, repeats=1, repeatDelay=0}
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Connecting
> as user 'hrt_qa'
> 2023-01-24T21:36:22,409 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.cbo.enable=true
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:22,619 INFO
[jira] [Work logged] (HIVE-26971) Backport of HIVE-20593
[
https://issues.apache.org/jira/browse/HIVE-26971?focusedWorklogId=842355=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842355
]
ASF GitHub Bot logged work on HIVE-26971:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 17:55
Start Date: 30/Jan/23 17:55
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3973:
URL: https://github.com/apache/hive/pull/3973#issuecomment-1409066466
@zabetak Can you please review this
Issue Time Tracking
---
Worklog Id: (was: 842355)
Time Spent: 40m (was: 0.5h)
> Backport of HIVE-20593
> --
>
> Key: HIVE-26971
> URL: https://issues.apache.org/jira/browse/HIVE-26971
> Project: Hive
> Issue Type: Sub-task
>Reporter: Aman Raj
>Assignee: Aman Raj
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> There are test failures are as follows :
> h4. Stacktrace
> java.lang.AssertionError:
> Client Execution succeeded but contained differences (error code = 1) after
> executing load_data_using_job.q
> 3031,3032d3030
> < PREHOOK: Input: default@orc_test_txn_{_}temp_table_for_load_data{_}_
> < PREHOOK: Output: default@orc_test_txn
> 3035d3032
> < POSTHOOK: Input: default@orc_test_txn_{_}temp_table_for_load_data{_}_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26898) Split Notification logging so that busy clusters can have better performance
[ https://issues.apache.org/jira/browse/HIVE-26898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682210#comment-17682210 ] Riju Trivedi commented on HIVE-26898: - [~tarak271] Do we see this notification_log table growing and OOM issues In recent versions 717, SP1 and SP2? Improvements like loading notifications in batches (HIVE-19430) and handling batches in different transactions (HIVE-24432) are already included in these versions. > Split Notification logging so that busy clusters can have better performance > > > Key: HIVE-26898 > URL: https://issues.apache.org/jira/browse/HIVE-26898 > Project: Hive > Issue Type: New Feature >Reporter: Taraka Rama Rao Lethavadla >Priority: Major > > For DDL & DML events are logged into notifications log table and those get > cleaned as soon as ttl got expired. > In most of the busy clusters, the notification log is growing even though > cleaner is running and kept on cleaning the events. It means the rate of Hive > db operations are very high compared to rate at which cleaning is happening. > So any query on this table is becoming bottle neck at backend DB causing slow > response > The proposal is to split the notification log table in to multiple tables > like > notification_log_dml - for all DML queries > notification_log_insert - for all insert queries > .. > etc. > > So that load on that single table gets reduced improving the performance of > the backend db as well as Hive -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26974) Iceberg: CTL from iceberg table should copy partition fields correctly
[ https://issues.apache.org/jira/browse/HIVE-26974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682207#comment-17682207 ] Ayush Saxena commented on HIVE-26974: - Committed to master. Thanx [~rameshkumar] for the review!!! > Iceberg: CTL from iceberg table should copy partition fields correctly > -- > > Key: HIVE-26974 > URL: https://issues.apache.org/jira/browse/HIVE-26974 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > # Create iceberg table. Ensure it to have a partition field. > # run "create external table like x" > # Created table in #2 misses out on creating relevant partition field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26974) Iceberg: CTL from iceberg table should copy partition fields correctly
[ https://issues.apache.org/jira/browse/HIVE-26974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-26974. - Fix Version/s: 4.0.0 Resolution: Fixed > Iceberg: CTL from iceberg table should copy partition fields correctly > -- > > Key: HIVE-26974 > URL: https://issues.apache.org/jira/browse/HIVE-26974 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > # Create iceberg table. Ensure it to have a partition field. > # run "create external table like x" > # Created table in #2 misses out on creating relevant partition field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26974) Iceberg: CTL from iceberg table should copy partition fields correctly
[ https://issues.apache.org/jira/browse/HIVE-26974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-26974: Summary: Iceberg: CTL from iceberg table should copy partition fields correctly (was: CTL from iceberg table should copy partition fields correctly) > Iceberg: CTL from iceberg table should copy partition fields correctly > -- > > Key: HIVE-26974 > URL: https://issues.apache.org/jira/browse/HIVE-26974 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > # Create iceberg table. Ensure it to have a partition field. > # run "create external table like x" > # Created table in #2 misses out on creating relevant partition field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26974) CTL from iceberg table should copy partition fields correctly
[ https://issues.apache.org/jira/browse/HIVE-26974?focusedWorklogId=842348=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842348 ] ASF GitHub Bot logged work on HIVE-26974: - Author: ASF GitHub Bot Created on: 30/Jan/23 17:41 Start Date: 30/Jan/23 17:41 Worklog Time Spent: 10m Work Description: ayushtkn merged PR #3980: URL: https://github.com/apache/hive/pull/3980 Issue Time Tracking --- Worklog Id: (was: 842348) Time Spent: 1h 10m (was: 1h) > CTL from iceberg table should copy partition fields correctly > - > > Key: HIVE-26974 > URL: https://issues.apache.org/jira/browse/HIVE-26974 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > # Create iceberg table. Ensure it to have a partition field. > # run "create external table like x" > # Created table in #2 misses out on creating relevant partition field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26948) Backport HIVE-21456 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-26948?focusedWorklogId=842340=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842340 ] ASF GitHub Bot logged work on HIVE-26948: - Author: ASF GitHub Bot Created on: 30/Jan/23 17:21 Start Date: 30/Jan/23 17:21 Worklog Time Spent: 10m Work Description: vihangk1 commented on PR #3989: URL: https://github.com/apache/hive/pull/3989#issuecomment-1409018555 Thanks @zabetak, I am not sure if @sourabh912 is actively monitoring PRs in Hive, but I can reach out to him. Since this is not really a new code but a backport of the master branch, I feel this is still pretty low risk. Almost all the new code in this PR is under a config flag which is disabled by default. Additionally, @amanraj2520 confirmed that all the tests which are failing are known test failures. Issue Time Tracking --- Worklog Id: (was: 842340) Time Spent: 1h 20m (was: 1h 10m) > Backport HIVE-21456 to branch-3 > --- > > Key: HIVE-26948 > URL: https://issues.apache.org/jira/browse/HIVE-26948 > Project: Hive > Issue Type: Improvement > Components: Metastore, Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Blocker > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > HIVE-21456 adds support to connect to Hive metastore over http transport. > This is a very useful feature especially in cloud based environments. > Creating this ticket to backport it to branch-3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26974) CTL from iceberg table should copy partition fields correctly
[ https://issues.apache.org/jira/browse/HIVE-26974?focusedWorklogId=842336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842336 ] ASF GitHub Bot logged work on HIVE-26974: - Author: ASF GitHub Bot Created on: 30/Jan/23 17:16 Start Date: 30/Jan/23 17:16 Worklog Time Spent: 10m Work Description: ramesh0201 commented on PR #3980: URL: https://github.com/apache/hive/pull/3980#issuecomment-1409010731 +1. LGTM Issue Time Tracking --- Worklog Id: (was: 842336) Time Spent: 1h (was: 50m) > CTL from iceberg table should copy partition fields correctly > - > > Key: HIVE-26974 > URL: https://issues.apache.org/jira/browse/HIVE-26974 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Rajesh Balamohan >Assignee: Ayush Saxena >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > # Create iceberg table. Ensure it to have a partition field. > # run "create external table like x" > # Created table in #2 misses out on creating relevant partition field. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26999) Upgrade MySQL Connector Java due to security CVEs
[ https://issues.apache.org/jira/browse/HIVE-26999?focusedWorklogId=842325=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842325 ] ASF GitHub Bot logged work on HIVE-26999: - Author: ASF GitHub Bot Created on: 30/Jan/23 16:33 Start Date: 30/Jan/23 16:33 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3996: URL: https://github.com/apache/hive/pull/3996#issuecomment-1408949270 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3996) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3996=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3996=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3996=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3996=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3996=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842325) Time Spent: 0.5h (was: 20m) > Upgrade MySQL Connector Java due to security CVEs > -- > > Key: HIVE-26999 > URL: https://issues.apache.org/jira/browse/HIVE-26999 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Attachments: tree.txt > > Time Spent: 0.5h > Remaining Estimate: 0h > > The following CVEs impact older versions of [MySQL Connector > Java|https://mvnrepository.com/artifact/mysql/mysql-connector-java] > * *CVE-2021-3711* : Critical - Impacts all versions up to (including) > 8.0.27 (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-3711]) > * *CVE-2021-3712* - High - Impacts all versions up to (including) 8.0.27 > (ref: > [https://nvd.nist.gov/vuln/detail/CVE-2021-37112)|https://nvd.nist.gov/vuln/detail/CVE-2021-3711] > * *CVE-2021-44531* - High - Impacts all versions up to (including) 8.0.28 > (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-44531]) > * *CVE-2022-21824* - High - Impacts all versions up to (including) 8.0.28 > (ref:[https://nvd.nist.gov/vuln/detail/CVE-2022-21824)] > Recommendation: *Upgrade* [*MySQL Connector > Java*|https://mvnrepository.com/artifact/mysql/mysql-connector-java] *to* > [*8.0.31*|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.31] > *or above* -- This
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842312=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842312
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 15:25
Start Date: 30/Jan/23 15:25
Worklog Time Spent: 10m
Work Description: VenuReddy2103 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090773527
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
+ }
+
+ private Long getDataStoreId(Class modelClass) throws MetaException {
+ExecutionContext ec = ((JDOPersistenceManager) pm).getExecutionContext();
+AbstractClassMetaData cmd =
ec.getMetaDataManager().getMetaDataForClass(modelClass,
ec.getClassLoaderResolver());
+if (cmd.getIdentityType() == IdentityType.DATASTORE) {
+ return (Long) ec.getStoreManager().getValueGenerationStrategyValue(ec,
cmd, -1);
+} else {
+ throw new MetaException("Identity type is not datastore.");
+}
+ }
+
+ private void insertInBatch(String tableName, String columns, int
columnCount, int rowCount,
+ BatchExecutionContext batchExecutionContext) throws MetaException {
+if (rowCount == 0 || columnCount == 0) {
+ return;
+}
+int maxRowsInBatch = (batchSize == NO_BATCHING) ? rowCount : batchSize;
+int maxBatches = rowCount / maxRowsInBatch;
+int last = rowCount % maxRowsInBatch;
+String rowFormat = "(" + repeat(",?", columnCount).substring(1) + ")";
+String query = "";
+if (maxBatches > 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat,
maxRowsInBatch);
+}
+int batchParamCount = maxRowsInBatch * columnCount;
+for (int batch = 0; batch < maxBatches; batch++) {
+ batchExecutionContext.execute(query, maxRowsInBatch, batchParamCount);
+}
+if (last != 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat, last);
+ batchExecutionContext.execute(query, last, last * columnCount);
+}
+ }
+
+ private void executeQuery(String
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842311
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 15:16
Start Date: 30/Jan/23 15:16
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3905:
URL: https://github.com/apache/hive/pull/3905#issuecomment-1408808173
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3905)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3905=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3905=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3905=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=CODE_SMELL)
[5 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3905=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3905=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3905=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 842311)
Time Spent: 6h 50m (was: 6h 40m)
> Explore moving to directsql for ObjectStore::addPartitions
> --
>
> Key: HIVE-26035
> URL: https://issues.apache.org/jira/browse/HIVE-26035
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Venugopal Reddy K
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> Currently {{addPartitions}} uses datanuclues and is super slow for large
> number of partitions. It will be good to move to direct sql. Lots of repeated
> SQLs can be avoided as well (e.g SDS, SERDE, TABLE_PARAMS)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842310=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842310
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 15:13
Start Date: 30/Jan/23 15:13
Worklog Time Spent: 10m
Work Description: VenuReddy2103 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090755549
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
+ }
+
+ private Long getDataStoreId(Class modelClass) throws MetaException {
+ExecutionContext ec = ((JDOPersistenceManager) pm).getExecutionContext();
+AbstractClassMetaData cmd =
ec.getMetaDataManager().getMetaDataForClass(modelClass,
ec.getClassLoaderResolver());
+if (cmd.getIdentityType() == IdentityType.DATASTORE) {
+ return (Long) ec.getStoreManager().getValueGenerationStrategyValue(ec,
cmd, -1);
+} else {
+ throw new MetaException("Identity type is not datastore.");
+}
+ }
+
+ private void insertInBatch(String tableName, String columns, int
columnCount, int rowCount,
+ BatchExecutionContext batchExecutionContext) throws MetaException {
+if (rowCount == 0 || columnCount == 0) {
+ return;
+}
+int maxRowsInBatch = (batchSize == NO_BATCHING) ? rowCount : batchSize;
+int maxBatches = rowCount / maxRowsInBatch;
+int last = rowCount % maxRowsInBatch;
+String rowFormat = "(" + repeat(",?", columnCount).substring(1) + ")";
+String query = "";
+if (maxBatches > 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat,
maxRowsInBatch);
+}
+int batchParamCount = maxRowsInBatch * columnCount;
+for (int batch = 0; batch < maxBatches; batch++) {
+ batchExecutionContext.execute(query, maxRowsInBatch, batchParamCount);
+}
+if (last != 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat, last);
+ batchExecutionContext.execute(query, last, last * columnCount);
+}
+ }
+
+ private void executeQuery(String
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842309=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842309
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 15:13
Start Date: 30/Jan/23 15:13
Worklog Time Spent: 10m
Work Description: VenuReddy2103 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090754911
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
+ }
+
+ private Long getDataStoreId(Class modelClass) throws MetaException {
+ExecutionContext ec = ((JDOPersistenceManager) pm).getExecutionContext();
+AbstractClassMetaData cmd =
ec.getMetaDataManager().getMetaDataForClass(modelClass,
ec.getClassLoaderResolver());
+if (cmd.getIdentityType() == IdentityType.DATASTORE) {
+ return (Long) ec.getStoreManager().getValueGenerationStrategyValue(ec,
cmd, -1);
+} else {
+ throw new MetaException("Identity type is not datastore.");
+}
+ }
+
+ private void insertInBatch(String tableName, String columns, int
columnCount, int rowCount,
+ BatchExecutionContext batchExecutionContext) throws MetaException {
+if (rowCount == 0 || columnCount == 0) {
+ return;
+}
+int maxRowsInBatch = (batchSize == NO_BATCHING) ? rowCount : batchSize;
+int maxBatches = rowCount / maxRowsInBatch;
+int last = rowCount % maxRowsInBatch;
+String rowFormat = "(" + repeat(",?", columnCount).substring(1) + ")";
+String query = "";
+if (maxBatches > 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat,
maxRowsInBatch);
+}
+int batchParamCount = maxRowsInBatch * columnCount;
+for (int batch = 0; batch < maxBatches; batch++) {
+ batchExecutionContext.execute(query, maxRowsInBatch, batchParamCount);
+}
+if (last != 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat, last);
+ batchExecutionContext.execute(query, last, last * columnCount);
+}
+ }
+
+ private void executeQuery(String
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842308=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842308
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 15:12
Start Date: 30/Jan/23 15:12
Worklog Time Spent: 10m
Work Description: VenuReddy2103 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090753168
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
+ }
+
+ private Long getDataStoreId(Class modelClass) throws MetaException {
+ExecutionContext ec = ((JDOPersistenceManager) pm).getExecutionContext();
+AbstractClassMetaData cmd =
ec.getMetaDataManager().getMetaDataForClass(modelClass,
ec.getClassLoaderResolver());
+if (cmd.getIdentityType() == IdentityType.DATASTORE) {
+ return (Long) ec.getStoreManager().getValueGenerationStrategyValue(ec,
cmd, -1);
+} else {
+ throw new MetaException("Identity type is not datastore.");
+}
+ }
+
+ private void insertInBatch(String tableName, String columns, int
columnCount, int rowCount,
+ BatchExecutionContext batchExecutionContext) throws MetaException {
+if (rowCount == 0 || columnCount == 0) {
+ return;
+}
+int maxRowsInBatch = (batchSize == NO_BATCHING) ? rowCount : batchSize;
+int maxBatches = rowCount / maxRowsInBatch;
+int last = rowCount % maxRowsInBatch;
+String rowFormat = "(" + repeat(",?", columnCount).substring(1) + ")";
+String query = "";
+if (maxBatches > 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat,
maxRowsInBatch);
+}
+int batchParamCount = maxRowsInBatch * columnCount;
+for (int batch = 0; batch < maxBatches; batch++) {
+ batchExecutionContext.execute(query, maxRowsInBatch, batchParamCount);
+}
+if (last != 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat, last);
+ batchExecutionContext.execute(query, last, last * columnCount);
+}
+ }
+
+ private void executeQuery(String
[jira] [Work logged] (HIVE-26996) typos in hive-exec
[ https://issues.apache.org/jira/browse/HIVE-26996?focusedWorklogId=842296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842296 ] ASF GitHub Bot logged work on HIVE-26996: - Author: ASF GitHub Bot Created on: 30/Jan/23 13:42 Start Date: 30/Jan/23 13:42 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3994: URL: https://github.com/apache/hive/pull/3994#issuecomment-1408652495 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3994) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3994=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3994=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3994=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=CODE_SMELL) [4 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3994=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3994=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3994=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842296) Time Spent: 20m (was: 10m) > typos in hive-exec > -- > > Key: HIVE-26996 > URL: https://issues.apache.org/jira/browse/HIVE-26996 > Project: Hive > Issue Type: Improvement > Components: Query Planning, Query Processor >Affects Versions: All Versions >Reporter: Michal Lorek >Assignee: Michal Lorek >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > typos and grammar errors in hive-exec module -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842282
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 12:59
Start Date: 30/Jan/23 12:59
Worklog Time Spent: 10m
Work Description: VenuReddy2103 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090594571
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
Review Comment:
done
Issue Time Tracking
---
Worklog Id: (was: 842282)
Time Spent: 6h 10m (was: 6h)
> Explore moving to directsql for ObjectStore::addPartitions
> --
>
> Key: HIVE-26035
> URL: https://issues.apache.org/jira/browse/HIVE-26035
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Venugopal Reddy K
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Currently {{addPartitions}} uses datanuclues and is super slow for large
> number of partitions. It will be good to move to direct sql. Lots of repeated
> SQLs can be avoided as well (e.g SDS, SERDE, TABLE_PARAMS)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842267
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 12:20
Start Date: 30/Jan/23 12:20
Worklog Time Spent: 10m
Work Description: VenuReddy2103 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090195330
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,835 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
+ }
+
+ private Long getDataStoreId(Class modelClass) throws MetaException {
+ExecutionContext ec = ((JDOPersistenceManager) pm).getExecutionContext();
+AbstractClassMetaData cmd =
ec.getMetaDataManager().getMetaDataForClass(modelClass,
ec.getClassLoaderResolver());
+if (cmd.getIdentityType() == IdentityType.DATASTORE) {
+ return (Long) ec.getStoreManager().getValueGenerationStrategyValue(ec,
cmd, -1);
+} else {
+ throw new MetaException("Identity type is not datastore.");
+}
+ }
+
+ private void insertInBatch(String tableName, String columns, int
columnCount, int rowCount,
+ BatchExecutionContext bec) throws MetaException {
+if (rowCount == 0 || columnCount == 0) {
+ return;
+}
+int maxRowsInBatch = (batchSize == NO_BATCHING) ? rowCount : batchSize;
+int maxBatches = rowCount / maxRowsInBatch;
+int last = rowCount % maxRowsInBatch;
+String rowFormat = "(" + repeat(",?", columnCount).substring(1) + ")";
+String query = "";
+if (maxBatches > 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat,
maxRowsInBatch);
+}
+int batchParamCount = maxRowsInBatch * columnCount;
+for (int batch = 0; batch < maxBatches; batch++) {
+ bec.execute(query, maxRowsInBatch, batchParamCount);
Review Comment:
done. Have removed this variable as part of another comment.
Issue Time Tracking
---
Worklog Id: (was: 842267)
Time Spent: 6h (was: 5h 50m)
> Explore moving to directsql for ObjectStore::addPartitions
>
[jira] [Updated] (HIVE-26999) Upgrade MySQL Connector Java due to security CVEs
[ https://issues.apache.org/jira/browse/HIVE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaspati Krishnatri updated HIVE-26999: Attachment: tree.txt > Upgrade MySQL Connector Java due to security CVEs > -- > > Key: HIVE-26999 > URL: https://issues.apache.org/jira/browse/HIVE-26999 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Attachments: tree.txt > > Time Spent: 20m > Remaining Estimate: 0h > > The following CVEs impact older versions of [MySQL Connector > Java|https://mvnrepository.com/artifact/mysql/mysql-connector-java] > * *CVE-2021-3711* : Critical - Impacts all versions up to (including) > 8.0.27 (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-3711]) > * *CVE-2021-3712* - High - Impacts all versions up to (including) 8.0.27 > (ref: > [https://nvd.nist.gov/vuln/detail/CVE-2021-37112)|https://nvd.nist.gov/vuln/detail/CVE-2021-3711] > * *CVE-2021-44531* - High - Impacts all versions up to (including) 8.0.28 > (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-44531]) > * *CVE-2022-21824* - High - Impacts all versions up to (including) 8.0.28 > (ref:[https://nvd.nist.gov/vuln/detail/CVE-2022-21824)] > Recommendation: *Upgrade* [*MySQL Connector > Java*|https://mvnrepository.com/artifact/mysql/mysql-connector-java] *to* > [*8.0.31*|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.31] > *or above* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26999) Upgrade MySQL Connector Java due to security CVEs
[ https://issues.apache.org/jira/browse/HIVE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaspati Krishnatri updated HIVE-26999: Attachment: (was: tree.txt) > Upgrade MySQL Connector Java due to security CVEs > -- > > Key: HIVE-26999 > URL: https://issues.apache.org/jira/browse/HIVE-26999 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Attachments: tree.txt > > Time Spent: 20m > Remaining Estimate: 0h > > The following CVEs impact older versions of [MySQL Connector > Java|https://mvnrepository.com/artifact/mysql/mysql-connector-java] > * *CVE-2021-3711* : Critical - Impacts all versions up to (including) > 8.0.27 (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-3711]) > * *CVE-2021-3712* - High - Impacts all versions up to (including) 8.0.27 > (ref: > [https://nvd.nist.gov/vuln/detail/CVE-2021-37112)|https://nvd.nist.gov/vuln/detail/CVE-2021-3711] > * *CVE-2021-44531* - High - Impacts all versions up to (including) 8.0.28 > (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-44531]) > * *CVE-2022-21824* - High - Impacts all versions up to (including) 8.0.28 > (ref:[https://nvd.nist.gov/vuln/detail/CVE-2022-21824)] > Recommendation: *Upgrade* [*MySQL Connector > Java*|https://mvnrepository.com/artifact/mysql/mysql-connector-java] *to* > [*8.0.31*|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.31] > *or above* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26999) Upgrade MySQL Connector Java due to security CVEs
[ https://issues.apache.org/jira/browse/HIVE-26999?focusedWorklogId=842260=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842260 ] ASF GitHub Bot logged work on HIVE-26999: - Author: ASF GitHub Bot Created on: 30/Jan/23 11:23 Start Date: 30/Jan/23 11:23 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3996: URL: https://github.com/apache/hive/pull/3996#issuecomment-1408448798 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3996) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3996=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3996=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3996=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3996=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3996=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3996=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842260) Time Spent: 20m (was: 10m) > Upgrade MySQL Connector Java due to security CVEs > -- > > Key: HIVE-26999 > URL: https://issues.apache.org/jira/browse/HIVE-26999 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Attachments: tree.txt > > Time Spent: 20m > Remaining Estimate: 0h > > The following CVEs impact older versions of [MySQL Connector > Java|https://mvnrepository.com/artifact/mysql/mysql-connector-java] > * *CVE-2021-3711* : Critical - Impacts all versions up to (including) > 8.0.27 (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-3711]) > * *CVE-2021-3712* - High - Impacts all versions up to (including) 8.0.27 > (ref: > [https://nvd.nist.gov/vuln/detail/CVE-2021-37112)|https://nvd.nist.gov/vuln/detail/CVE-2021-3711] > * *CVE-2021-44531* - High - Impacts all versions up to (including) 8.0.28 > (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-44531]) > * *CVE-2022-21824* - High - Impacts all versions up to (including) 8.0.28 > (ref:[https://nvd.nist.gov/vuln/detail/CVE-2022-21824)] > Recommendation: *Upgrade* [*MySQL Connector > Java*|https://mvnrepository.com/artifact/mysql/mysql-connector-java] *to* > [*8.0.31*|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.31] > *or above* -- This
[jira] [Commented] (HIVE-26995) Iceberg: Enhance time travel syntax with expressions
[
https://issues.apache.org/jira/browse/HIVE-26995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682004#comment-17682004
]
Denys Kuzmenko commented on HIVE-26995:
---
Merged to master
[~krisztiankasa], thank you for the review!
> Iceberg: Enhance time travel syntax with expressions
>
>
> Key: HIVE-26995
> URL: https://issues.apache.org/jira/browse/HIVE-26995
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Allow expressions in time travel queries, such as
> {code}
> FOR SYSTEM_TIME AS OF CURRENT_TIMESTAMP - interval '10' hours
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26995) Iceberg: Enhance time travel syntax with expressions
[
https://issues.apache.org/jira/browse/HIVE-26995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko resolved HIVE-26995.
---
Fix Version/s: 4.0.0
Resolution: Fixed
> Iceberg: Enhance time travel syntax with expressions
>
>
> Key: HIVE-26995
> URL: https://issues.apache.org/jira/browse/HIVE-26995
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Allow expressions in time travel queries, such as
> {code}
> FOR SYSTEM_TIME AS OF CURRENT_TIMESTAMP - interval '10' hours
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26995) Iceberg: Enhance time travel syntax with expressions
[
https://issues.apache.org/jira/browse/HIVE-26995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko reassigned HIVE-26995:
-
Assignee: Denys Kuzmenko
> Iceberg: Enhance time travel syntax with expressions
>
>
> Key: HIVE-26995
> URL: https://issues.apache.org/jira/browse/HIVE-26995
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Allow expressions in time travel queries, such as
> {code}
> FOR SYSTEM_TIME AS OF CURRENT_TIMESTAMP - interval '10' hours
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26935) Expose root cause of MetaException to client sides
[
https://issues.apache.org/jira/browse/HIVE-26935?focusedWorklogId=842237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842237
]
ASF GitHub Bot logged work on HIVE-26935:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 10:33
Start Date: 30/Jan/23 10:33
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3938:
URL: https://github.com/apache/hive/pull/3938#discussion_r1090441719
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java:
##
@@ -203,11 +203,12 @@ public Result invokeInternal(final Object proxy, final
Method method, final Obje
}
}
- if (retryCount >= retryLimit) {
+ Throwable rootCause = ExceptionUtils.getRootCause(caughtException);
+ String errorMessage = ExceptionUtils.getMessage(caughtException) +
+ (rootCause == null ? "" : ("\nRoot cause: " + rootCause));
+ if (!RetryingMetaStoreClient.isRecoverableMessage(errorMessage) ||
retryCount >= retryLimit) {
LOG.error("HMSHandler Fatal error: " +
ExceptionUtils.getStackTrace(caughtException));
-MetaException me = new MetaException(caughtException.toString());
-me.initCause(caughtException);
Review Comment:
If this change was somewhere deeper in the metastore code then I would
probably keep the code as is.
In this case though, we are just before returning the exception to the
client, I finally agree with you that using `initCause` is a bit useless.
Issue Time Tracking
---
Worklog Id: (was: 842237)
Time Spent: 3h (was: 2h 50m)
> Expose root cause of MetaException to client sides
> --
>
> Key: HIVE-26935
> URL: https://issues.apache.org/jira/browse/HIVE-26935
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h
> Remaining Estimate: 0h
>
> MetaException is generated by thrift, and only {{message}} filed will be
> transport to client, we should expose the root cause in message to the
> clients with following advantages:
> * More friendly for user troubleshooting
> * Some root cause is unrecoverable, exposing it can disable the unnecessary
> retry.
> *How to Reproduce:*
> - Step 1: Disable direct sql for HMS for our test case.
> - Step 2: Add an illegal {{PART_COL_STATS}} for a partition,
> - Step 3: Try to {{drop table}} with Spark.
> The exception in Hive metastore is:
> {code:sh}
> 2023-01-11T17:13:51,259 ERROR [Metastore-Handler-Pool: Thread-39]:
> metastore.ObjectStore (ObjectStore.java:run(4369)) -
> javax.jdo.JDOUserException: One or more instances could not be deleted
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:625)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.datanucleus.api.jdo.JDOQuery.deletePersistentInternal(JDOQuery.java:530)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.datanucleus.api.jdo.JDOQuery.deletePersistentAll(JDOQuery.java:499)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.apache.hadoop.hive.metastore.QueryWrapper.deletePersistentAll(QueryWrapper.java:108)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsNoTxn(ObjectStore.java:4207)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:285)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$7.run(ObjectStore.java:3086)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:74)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsViaJdo(ObjectStore.java:3074)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$400(ObjectStore.java:285)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3058)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3050)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4362)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-26999) Upgrade MySQL Connector Java due to security CVEs
[ https://issues.apache.org/jira/browse/HIVE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26999: -- Labels: pull-request-available (was: ) > Upgrade MySQL Connector Java due to security CVEs > -- > > Key: HIVE-26999 > URL: https://issues.apache.org/jira/browse/HIVE-26999 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Attachments: tree.txt > > Time Spent: 10m > Remaining Estimate: 0h > > The following CVEs impact older versions of [MySQL Connector > Java|https://mvnrepository.com/artifact/mysql/mysql-connector-java] > * *CVE-2021-3711* : Critical - Impacts all versions up to (including) > 8.0.27 (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-3711]) > * *CVE-2021-3712* - High - Impacts all versions up to (including) 8.0.27 > (ref: > [https://nvd.nist.gov/vuln/detail/CVE-2021-37112)|https://nvd.nist.gov/vuln/detail/CVE-2021-3711] > * *CVE-2021-44531* - High - Impacts all versions up to (including) 8.0.28 > (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-44531]) > * *CVE-2022-21824* - High - Impacts all versions up to (including) 8.0.28 > (ref:[https://nvd.nist.gov/vuln/detail/CVE-2022-21824)] > Recommendation: *Upgrade* [*MySQL Connector > Java*|https://mvnrepository.com/artifact/mysql/mysql-connector-java] *to* > [*8.0.31*|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.31] > *or above* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26999) Upgrade MySQL Connector Java due to security CVEs
[ https://issues.apache.org/jira/browse/HIVE-26999?focusedWorklogId=842235=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842235 ] ASF GitHub Bot logged work on HIVE-26999: - Author: ASF GitHub Bot Created on: 30/Jan/23 10:30 Start Date: 30/Jan/23 10:30 Worklog Time Spent: 10m Work Description: devaspatikrishnatri opened a new pull request, #3996: URL: https://github.com/apache/hive/pull/3996 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Issue Time Tracking --- Worklog Id: (was: 842235) Remaining Estimate: 0h Time Spent: 10m > Upgrade MySQL Connector Java due to security CVEs > -- > > Key: HIVE-26999 > URL: https://issues.apache.org/jira/browse/HIVE-26999 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Attachments: tree.txt > > Time Spent: 10m > Remaining Estimate: 0h > > The following CVEs impact older versions of [MySQL Connector > Java|https://mvnrepository.com/artifact/mysql/mysql-connector-java] > * *CVE-2021-3711* : Critical - Impacts all versions up to (including) > 8.0.27 (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-3711]) > * *CVE-2021-3712* - High - Impacts all versions up to (including) 8.0.27 > (ref: > [https://nvd.nist.gov/vuln/detail/CVE-2021-37112)|https://nvd.nist.gov/vuln/detail/CVE-2021-3711] > * *CVE-2021-44531* - High - Impacts all versions up to (including) 8.0.28 > (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-44531]) > * *CVE-2022-21824* - High - Impacts all versions up to (including) 8.0.28 > (ref:[https://nvd.nist.gov/vuln/detail/CVE-2022-21824)] > Recommendation: *Upgrade* [*MySQL Connector > Java*|https://mvnrepository.com/artifact/mysql/mysql-connector-java] *to* > [*8.0.31*|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.31] > *or above* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26999) Upgrade MySQL Connector Java due to security CVEs
[ https://issues.apache.org/jira/browse/HIVE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaspati Krishnatri updated HIVE-26999: Attachment: tree.txt > Upgrade MySQL Connector Java due to security CVEs > -- > > Key: HIVE-26999 > URL: https://issues.apache.org/jira/browse/HIVE-26999 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Attachments: tree.txt > > > The following CVEs impact older versions of [MySQL Connector > Java|https://mvnrepository.com/artifact/mysql/mysql-connector-java] > * *CVE-2021-3711* : Critical - Impacts all versions up to (including) > 8.0.27 (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-3711]) > * *CVE-2021-3712* - High - Impacts all versions up to (including) 8.0.27 > (ref: > [https://nvd.nist.gov/vuln/detail/CVE-2021-37112)|https://nvd.nist.gov/vuln/detail/CVE-2021-3711] > * *CVE-2021-44531* - High - Impacts all versions up to (including) 8.0.28 > (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-44531]) > * *CVE-2022-21824* - High - Impacts all versions up to (including) 8.0.28 > (ref:[https://nvd.nist.gov/vuln/detail/CVE-2022-21824)] > Recommendation: *Upgrade* [*MySQL Connector > Java*|https://mvnrepository.com/artifact/mysql/mysql-connector-java] *to* > [*8.0.31*|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.31] > *or above* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26999) Upgrade MySQL Connector Java due to security CVEs
[ https://issues.apache.org/jira/browse/HIVE-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaspati Krishnatri reassigned HIVE-26999: --- Assignee: Devaspati Krishnatri > Upgrade MySQL Connector Java due to security CVEs > -- > > Key: HIVE-26999 > URL: https://issues.apache.org/jira/browse/HIVE-26999 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > > The following CVEs impact older versions of [MySQL Connector > Java|https://mvnrepository.com/artifact/mysql/mysql-connector-java] > * *CVE-2021-3711* : Critical - Impacts all versions up to (including) > 8.0.27 (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-3711]) > * *CVE-2021-3712* - High - Impacts all versions up to (including) 8.0.27 > (ref: > [https://nvd.nist.gov/vuln/detail/CVE-2021-37112)|https://nvd.nist.gov/vuln/detail/CVE-2021-3711] > * *CVE-2021-44531* - High - Impacts all versions up to (including) 8.0.28 > (ref: [https://nvd.nist.gov/vuln/detail/CVE-2021-44531]) > * *CVE-2022-21824* - High - Impacts all versions up to (including) 8.0.28 > (ref:[https://nvd.nist.gov/vuln/detail/CVE-2022-21824)] > Recommendation: *Upgrade* [*MySQL Connector > Java*|https://mvnrepository.com/artifact/mysql/mysql-connector-java] *to* > [*8.0.31*|https://mvnrepository.com/artifact/mysql/mysql-connector-java/8.0.31] > *or above* -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26935) Expose root cause of MetaException to client sides
[
https://issues.apache.org/jira/browse/HIVE-26935?focusedWorklogId=842228=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842228
]
ASF GitHub Bot logged work on HIVE-26935:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 10:07
Start Date: 30/Jan/23 10:07
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3938:
URL: https://github.com/apache/hive/pull/3938#discussion_r1090412818
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RetryingHMSHandler.java:
##
@@ -203,11 +203,12 @@ public Result invokeInternal(final Object proxy, final
Method method, final Obje
}
}
- if (retryCount >= retryLimit) {
+ Throwable rootCause = ExceptionUtils.getRootCause(caughtException);
+ String errorMessage = ExceptionUtils.getMessage(caughtException) +
Review Comment:
This is an improvement given the current situation and that's why I think
its worth merging. My comments are more forward looking on things that we could
improve in the future.
Issue Time Tracking
---
Worklog Id: (was: 842228)
Time Spent: 2h 50m (was: 2h 40m)
> Expose root cause of MetaException to client sides
> --
>
> Key: HIVE-26935
> URL: https://issues.apache.org/jira/browse/HIVE-26935
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 4.0.0-alpha-2
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> MetaException is generated by thrift, and only {{message}} filed will be
> transport to client, we should expose the root cause in message to the
> clients with following advantages:
> * More friendly for user troubleshooting
> * Some root cause is unrecoverable, exposing it can disable the unnecessary
> retry.
> *How to Reproduce:*
> - Step 1: Disable direct sql for HMS for our test case.
> - Step 2: Add an illegal {{PART_COL_STATS}} for a partition,
> - Step 3: Try to {{drop table}} with Spark.
> The exception in Hive metastore is:
> {code:sh}
> 2023-01-11T17:13:51,259 ERROR [Metastore-Handler-Pool: Thread-39]:
> metastore.ObjectStore (ObjectStore.java:run(4369)) -
> javax.jdo.JDOUserException: One or more instances could not be deleted
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:625)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.datanucleus.api.jdo.JDOQuery.deletePersistentInternal(JDOQuery.java:530)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.datanucleus.api.jdo.JDOQuery.deletePersistentAll(JDOQuery.java:499)
> ~[datanucleus-api-jdo-5.2.8.jar:?]
> at
> org.apache.hadoop.hive.metastore.QueryWrapper.deletePersistentAll(QueryWrapper.java:108)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsNoTxn(ObjectStore.java:4207)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$1000(ObjectStore.java:285)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$7.run(ObjectStore.java:3086)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.Batchable.runBatched(Batchable.java:74)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsViaJdo(ObjectStore.java:3074)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.access$400(ObjectStore.java:285)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3058)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$6.getJdoResult(ObjectStore.java:3050)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:4362)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitionsInternal(ObjectStore.java:3061)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.dropPartitions(ObjectStore.java:3040)
> ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_332]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842225
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 09:52
Start Date: 30/Jan/23 09:52
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090385606
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
+ }
+
+ private Long getDataStoreId(Class modelClass) throws MetaException {
+ExecutionContext ec = ((JDOPersistenceManager) pm).getExecutionContext();
+AbstractClassMetaData cmd =
ec.getMetaDataManager().getMetaDataForClass(modelClass,
ec.getClassLoaderResolver());
+if (cmd.getIdentityType() == IdentityType.DATASTORE) {
+ return (Long) ec.getStoreManager().getValueGenerationStrategyValue(ec,
cmd, -1);
+} else {
+ throw new MetaException("Identity type is not datastore.");
+}
+ }
+
+ private void insertInBatch(String tableName, String columns, int
columnCount, int rowCount,
+ BatchExecutionContext batchExecutionContext) throws MetaException {
+if (rowCount == 0 || columnCount == 0) {
+ return;
+}
+int maxRowsInBatch = (batchSize == NO_BATCHING) ? rowCount : batchSize;
+int maxBatches = rowCount / maxRowsInBatch;
+int last = rowCount % maxRowsInBatch;
+String rowFormat = "(" + repeat(",?", columnCount).substring(1) + ")";
+String query = "";
+if (maxBatches > 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat,
maxRowsInBatch);
+}
+int batchParamCount = maxRowsInBatch * columnCount;
+for (int batch = 0; batch < maxBatches; batch++) {
+ batchExecutionContext.execute(query, maxRowsInBatch, batchParamCount);
+}
+if (last != 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat, last);
+ batchExecutionContext.execute(query, last, last * columnCount);
+}
+ }
+
+ private void executeQuery(String
[jira] [Work logged] (HIVE-26948) Backport HIVE-21456 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-26948?focusedWorklogId=842226=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842226 ] ASF GitHub Bot logged work on HIVE-26948: - Author: ASF GitHub Bot Created on: 30/Jan/23 09:52 Start Date: 30/Jan/23 09:52 Worklog Time Spent: 10m Work Description: zabetak commented on PR #3989: URL: https://github.com/apache/hive/pull/3989#issuecomment-1408302382 @vihangk1 I would prefer if @sourabh912 reviews the changes here since he is the author of the original patch; I will +1 if he does! In terms of process, I would prefer if we don't backport other changes till the build is stabilized; there are many times were failures seem unrelated to a specific PR but in practice they are not. Issue Time Tracking --- Worklog Id: (was: 842226) Time Spent: 1h 10m (was: 1h) > Backport HIVE-21456 to branch-3 > --- > > Key: HIVE-26948 > URL: https://issues.apache.org/jira/browse/HIVE-26948 > Project: Hive > Issue Type: Improvement > Components: Metastore, Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Blocker > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > HIVE-21456 adds support to connect to Hive metastore over http transport. > This is a very useful feature especially in cloud based environments. > Creating this ticket to backport it to branch-3. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=84=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-84
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 09:47
Start Date: 30/Jan/23 09:47
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090387208
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
+ }
+
+ private Long getDataStoreId(Class modelClass) throws MetaException {
+ExecutionContext ec = ((JDOPersistenceManager) pm).getExecutionContext();
+AbstractClassMetaData cmd =
ec.getMetaDataManager().getMetaDataForClass(modelClass,
ec.getClassLoaderResolver());
+if (cmd.getIdentityType() == IdentityType.DATASTORE) {
+ return (Long) ec.getStoreManager().getValueGenerationStrategyValue(ec,
cmd, -1);
+} else {
+ throw new MetaException("Identity type is not datastore.");
+}
+ }
+
+ private void insertInBatch(String tableName, String columns, int
columnCount, int rowCount,
+ BatchExecutionContext batchExecutionContext) throws MetaException {
+if (rowCount == 0 || columnCount == 0) {
+ return;
+}
+int maxRowsInBatch = (batchSize == NO_BATCHING) ? rowCount : batchSize;
+int maxBatches = rowCount / maxRowsInBatch;
+int last = rowCount % maxRowsInBatch;
+String rowFormat = "(" + repeat(",?", columnCount).substring(1) + ")";
+String query = "";
+if (maxBatches > 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat,
maxRowsInBatch);
+}
+int batchParamCount = maxRowsInBatch * columnCount;
+for (int batch = 0; batch < maxBatches; batch++) {
+ batchExecutionContext.execute(query, maxRowsInBatch, batchParamCount);
+}
+if (last != 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat, last);
+ batchExecutionContext.execute(query, last, last * columnCount);
+}
+ }
+
+ private void executeQuery(String
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842221
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 09:45
Start Date: 30/Jan/23 09:45
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090385606
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
+ }
+
+ private Long getDataStoreId(Class modelClass) throws MetaException {
+ExecutionContext ec = ((JDOPersistenceManager) pm).getExecutionContext();
+AbstractClassMetaData cmd =
ec.getMetaDataManager().getMetaDataForClass(modelClass,
ec.getClassLoaderResolver());
+if (cmd.getIdentityType() == IdentityType.DATASTORE) {
+ return (Long) ec.getStoreManager().getValueGenerationStrategyValue(ec,
cmd, -1);
+} else {
+ throw new MetaException("Identity type is not datastore.");
+}
+ }
+
+ private void insertInBatch(String tableName, String columns, int
columnCount, int rowCount,
+ BatchExecutionContext batchExecutionContext) throws MetaException {
+if (rowCount == 0 || columnCount == 0) {
+ return;
+}
+int maxRowsInBatch = (batchSize == NO_BATCHING) ? rowCount : batchSize;
+int maxBatches = rowCount / maxRowsInBatch;
+int last = rowCount % maxRowsInBatch;
+String rowFormat = "(" + repeat(",?", columnCount).substring(1) + ")";
+String query = "";
+if (maxBatches > 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat,
maxRowsInBatch);
+}
+int batchParamCount = maxRowsInBatch * columnCount;
+for (int batch = 0; batch < maxBatches; batch++) {
+ batchExecutionContext.execute(query, maxRowsInBatch, batchParamCount);
+}
+if (last != 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat, last);
+ batchExecutionContext.execute(query, last, last * columnCount);
+}
+ }
+
+ private void executeQuery(String
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842219
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 09:42
Start Date: 30/Jan/23 09:42
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090382021
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
+ }
+
+ private Long getDataStoreId(Class modelClass) throws MetaException {
+ExecutionContext ec = ((JDOPersistenceManager) pm).getExecutionContext();
+AbstractClassMetaData cmd =
ec.getMetaDataManager().getMetaDataForClass(modelClass,
ec.getClassLoaderResolver());
+if (cmd.getIdentityType() == IdentityType.DATASTORE) {
+ return (Long) ec.getStoreManager().getValueGenerationStrategyValue(ec,
cmd, -1);
+} else {
+ throw new MetaException("Identity type is not datastore.");
+}
+ }
+
+ private void insertInBatch(String tableName, String columns, int
columnCount, int rowCount,
+ BatchExecutionContext batchExecutionContext) throws MetaException {
+if (rowCount == 0 || columnCount == 0) {
+ return;
+}
+int maxRowsInBatch = (batchSize == NO_BATCHING) ? rowCount : batchSize;
+int maxBatches = rowCount / maxRowsInBatch;
+int last = rowCount % maxRowsInBatch;
+String rowFormat = "(" + repeat(",?", columnCount).substring(1) + ")";
+String query = "";
+if (maxBatches > 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat,
maxRowsInBatch);
+}
+int batchParamCount = maxRowsInBatch * columnCount;
+for (int batch = 0; batch < maxBatches; batch++) {
+ batchExecutionContext.execute(query, maxRowsInBatch, batchParamCount);
+}
+if (last != 0) {
+ query = dbType.getBatchInsertQuery(tableName, columns, rowFormat, last);
+ batchExecutionContext.execute(query, last, last * columnCount);
+}
+ }
+
+ private void executeQuery(String
[jira] [Work logged] (HIVE-26035) Explore moving to directsql for ObjectStore::addPartitions
[
https://issues.apache.org/jira/browse/HIVE-26035?focusedWorklogId=842217=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842217
]
ASF GitHub Bot logged work on HIVE-26035:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 09:42
Start Date: 30/Jan/23 09:42
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on code in PR #3905:
URL: https://github.com/apache/hive/pull/3905#discussion_r1090381771
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MFieldSchema;
+import org.apache.hadoop.hive.metastore.model.MOrder;
+import org.apache.hadoop.hive.metastore.model.MPartition;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnPrivilege;
+import org.apache.hadoop.hive.metastore.model.MPartitionPrivilege;
+import org.apache.hadoop.hive.metastore.model.MSerDeInfo;
+import org.apache.hadoop.hive.metastore.model.MStorageDescriptor;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.identity.DatastoreId;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+
+/**
+ * This class contains the methods to insert into tables on the underlying
database using direct SQL
+ */
+class DirectSqlInsertPart {
+ private final PersistenceManager pm;
+ private final DatabaseProduct dbType;
+ private final int batchSize;
+
+ public DirectSqlInsertPart(PersistenceManager pm, DatabaseProduct dbType,
int batchSize) {
+this.pm = pm;
+this.dbType = dbType;
+this.batchSize = batchSize;
+ }
+
+ /**
+ * Interface to execute multiple row insert query in batch for direct SQL
+ */
+ interface BatchExecutionContext {
+void execute(String batchQueryText, int batchRowCount, int
batchParamCount) throws MetaException;
Review Comment:
The `batchParamCount` is known from the caller context, can we remove the
parameter here?
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java:
##
@@ -0,0 +1,804 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import static org.apache.commons.lang3.StringUtils.repeat;
+import static org.apache.hadoop.hive.metastore.Batchable.NO_BATCHING;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+import javax.jdo.PersistenceManager;
+
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import
[jira] [Work logged] (HIVE-26995) Iceberg: Enhance time travel syntax with expressions
[
https://issues.apache.org/jira/browse/HIVE-26995?focusedWorklogId=842213=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842213
]
ASF GitHub Bot logged work on HIVE-26995:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 09:31
Start Date: 30/Jan/23 09:31
Worklog Time Spent: 10m
Work Description: deniskuzZ merged PR #3988:
URL: https://github.com/apache/hive/pull/3988
Issue Time Tracking
---
Worklog Id: (was: 842213)
Time Spent: 1h (was: 50m)
> Iceberg: Enhance time travel syntax with expressions
>
>
> Key: HIVE-26995
> URL: https://issues.apache.org/jira/browse/HIVE-26995
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Allow expressions in time travel queries, such as
> {code}
> FOR SYSTEM_TIME AS OF CURRENT_TIMESTAMP - interval '10' hours
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26980) Creation of materialized view stored by Iceberg fails if source table has tinyint column
[
https://issues.apache.org/jira/browse/HIVE-26980?focusedWorklogId=842194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842194
]
ASF GitHub Bot logged work on HIVE-26980:
-
Author: ASF GitHub Bot
Created on: 30/Jan/23 08:39
Start Date: 30/Jan/23 08:39
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3993:
URL: https://github.com/apache/hive/pull/3993#issuecomment-1408191855
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3993)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3993=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3993=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3993=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3993=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3993=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3993=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 842194)
Time Spent: 50m (was: 40m)
> Creation of materialized view stored by Iceberg fails if source table has
> tinyint column
>
>
> Key: HIVE-26980
> URL: https://issues.apache.org/jira/browse/HIVE-26980
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
>Reporter: Dharmik Thakkar
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> MV creation using iceberg storage format for Hive table containing tinyint
> column fails with error "java.lang.ClassCastException: class java.lang.Byte
> cannot be cast to class java.lang.Integer"
> Logs
> {code:java}
> 2023-01-24T21:36:22,153 INFO [pool-2-thread-1] jdbc.TestDriver: Beginning
> Test at 2023-01-24 21:36:22,153
> 2023-01-24T21:36:22,154 INFO [pool-2-thread-1] jdbc.TestDriver: BEGIN MAIN
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Running
> SessionGroup{name='SG_TNMECIDQ31', initialDelay=0, repeats=1, repeatDelay=0}
> 2023-01-24T21:36:22,155 INFO [pool-38-thread-1] jdbc.TestDriver: Connecting
> as user 'hrt_qa'
> 2023-01-24T21:36:22,409 INFO [pool-38-thread-1] jdbc.TestDriver: Query: set
> hive.cbo.enable=true
> 2023-01-24T21:36:22,619 INFO [pool-38-thread-1] jdbc.TestDriver: No output
> to verify
> 2023-01-24T21:36:22,619 INFO
[jira] [Commented] (HIVE-17502) Reuse of default session should not throw an exception in LLAP w/ Tez
[
https://issues.apache.org/jira/browse/HIVE-17502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681951#comment-17681951
]
Stamatis Zampetakis commented on HIVE-17502:
Hey [~ajaykumardev36], I don't have much context around this issue and not
enough time to push this forward.
Note that the patch submission is currently done via pull requests on GitHub
(https://github.com/apache/hive/pulls). If you want to push this forward start
by raising a pull request. When tests pass try to find a reviewer by sending an
email to the dev list; note that we are quite short on reviewers the past few
months so it might not be trivial.
> Reuse of default session should not throw an exception in LLAP w/ Tez
> -
>
> Key: HIVE-17502
> URL: https://issues.apache.org/jira/browse/HIVE-17502
> Project: Hive
> Issue Type: Bug
> Components: llap, Tez
>Affects Versions: 2.1.1, 2.2.0
> Environment: HDP 2.6.1.0-129, Hue 4
>Reporter: Thai Bui
>Priority: Blocker
> Attachments: HIVE-17502.2.patch, HIVE-17502.3.patch, HIVE-17502.patch
>
>
> Hive2 w/ LLAP on Tez doesn't allow a currently used, default session to be
> skipped mostly because of this line
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java#L365.
> However, some clients such as Hue 4, allow multiple sessions to be used per
> user. Under this configuration, a Thrift client will send a request to either
> reuse or open a new session. The reuse request could include the session id
> of a currently used snippet being executed in Hue, this causes HS2 to throw
> an exception:
> {noformat}
> 2017-09-10T17:51:36,548 INFO [Thread-89]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(512)) - The current user:
> hive, session user: hive
> 2017-09-10T17:51:36,549 ERROR [Thread-89]: exec.Task
> (TezTask.java:execute(232)) - Failed to execute tez graph.
> org.apache.hadoop.hive.ql.metadata.HiveException: The pool session
> sessionId=5b61a578-6336-41c5-860d-9838166f97fe, queueName=llap, user=hive,
> doAs=false, isOpen=true, isDefault=true, expires in 591015330ms should have
> been returned to the pool
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.canWorkWithSameSession(TezSessionPoolManager.java:534)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.getSession(TezSessionPoolManager.java:544)
> ~[hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:147)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:79)
> [hive-exec-2.1.0.2.6.1.0-129.jar:2.1.0.2.6.1.0-129]
> {noformat}
> Note that every query is issued as a single 'hive' user to share the LLAP
> daemon pool, a set of pre-determined number of AMs is initialized at setup
> time. Thus, HS2 should allow new sessions from a Thrift client to be used out
> of the pool, or an existing session to be skipped and an unused session from
> the pool to be returned. The logic to throw an exception in the
> `canWorkWithSameSession` doesn't make sense to me.
> I have a solution to fix this issue in my local branch at
> https://github.com/thaibui/hive/commit/078a521b9d0906fe6c0323b63e567f6eee2f3a70.
> When applied, the log will become like so
> {noformat}
> 2017-09-10T09:15:33,578 INFO [Thread-239]: tez.TezSessionPoolManager
> (TezSessionPoolManager.java:canWorkWithSameSession(533)) - Skipping default
> session sessionId=6638b1da-0f8a-405e-85f0-9586f484e6de, queueName=llap,
> user=hive, doAs=false, isOpen=true, isDefault=true, expires in 591868732ms
> since it is being used.
> {noformat}
> A test case is provided in my branch to demonstrate how it works. If possible
> I would like this patch to be applied to version 2.1, 2.2 and master. Since
> we are using 2.1 LLAP in production with Hue 4, this patch is critical to our
> success.
> Alternatively, if this patch is too broad in scope, I propose adding an
> option to allow "skipping of currently used default sessions". With this new
> option default to "false", existing behavior won't change unless the option
> is turned on.
> I will prepare an official path if this change to master &/ the other
> branches is acceptable. I'm not an contributor &/ committer, this will be my
> first time contributing to
[jira] [Work logged] (HIVE-26960) Optimized bootstrap does not drop newly added tables at source.
[ https://issues.apache.org/jira/browse/HIVE-26960?focusedWorklogId=842189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-842189 ] ASF GitHub Bot logged work on HIVE-26960: - Author: ASF GitHub Bot Created on: 30/Jan/23 08:25 Start Date: 30/Jan/23 08:25 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3995: URL: https://github.com/apache/hive/pull/3995#issuecomment-1408175234 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3995) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3995=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3995=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3995=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3995=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3995=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3995=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 842189) Time Spent: 0.5h (was: 20m) > Optimized bootstrap does not drop newly added tables at source. > --- > > Key: HIVE-26960 > URL: https://issues.apache.org/jira/browse/HIVE-26960 > Project: Hive > Issue Type: Bug >Reporter: Rakshith C >Assignee: Rakshith C >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Scenario: > Replication is setup from DR to PROD after failover from PROD to DR and no > existing tables are modified at PROD but a new table is added at PROD. > Observations: > * _bootstrap directory won't be created during second cycle of optimized > bootstrap because existing tables were not modified. > * Based on this, it will not initialize list of tables to drop at PROD. > * This leads to the new table created at PROD not being dropped. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26996) typos in hive-exec
[ https://issues.apache.org/jira/browse/HIVE-26996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michal Lorek reassigned HIVE-26996: --- Assignee: Michal Lorek > typos in hive-exec > -- > > Key: HIVE-26996 > URL: https://issues.apache.org/jira/browse/HIVE-26996 > Project: Hive > Issue Type: Improvement > Components: Query Planning, Query Processor >Affects Versions: All Versions >Reporter: Michal Lorek >Assignee: Michal Lorek >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > typos and grammar errors in hive-exec module -- This message was sent by Atlassian Jira (v8.20.10#820010)
