[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[
https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=798412=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-798412
]
ASF GitHub Bot logged work on HIVE-24483:
-
Author: ASF GitHub Bot
Created on: 05/Aug/22 12:33
Start Date: 05/Aug/22 12:33
Worklog Time Spent: 10m
Work Description: ayushtkn commented on code in PR #3498:
URL: https://github.com/apache/hive/pull/3498#discussion_r938766882
##
standalone-metastore/metastore-common/pom.xml:
##
@@ -457,7 +457,7 @@
run
-
${protobuf.group}:protoc:${protobuf-exc.version}
+
com.google.protobuf:protoc:3.21.4
Review Comment:
Makes sense changed in the latest commit
Issue Time Tracking
---
Worklog Id: (was: 798412)
Time Spent: 3h 50m (was: 3h 40m)
> Bump protobuf version to 3.21.1 (aarch64 support)
> -
>
> Key: HIVE-24483
> URL: https://issues.apache.org/jira/browse/HIVE-24483
> Project: Hive
> Issue Type: Improvement
>Reporter: Cameron Moberg
>Assignee: Ayush Saxena
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> The following protoc version's used in hive is very old i.e. 2.5.0
> [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The
> v2.5.0 does not have aarch64 support. But the AArch64 support started from
> v3.5.0 on-words in google's protobuf project release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[ https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=798408=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-798408 ] ASF GitHub Bot logged work on HIVE-24483: - Author: ASF GitHub Bot Created on: 05/Aug/22 12:07 Start Date: 05/Aug/22 12:07 Worklog Time Spent: 10m Work Description: ayushtkn commented on code in PR #3498: URL: https://github.com/apache/hive/pull/3498#discussion_r938747643 ## data/files/datasets/src_hbase/load.hive.sql: ## @@ -1,3 +1,5 @@ +DROP table if exists src_hbase_tmp; Review Comment: Yeps, I am not sure, but my test failed once and in the next iteration when I fixed it, It started giving me AlreadyExist exception for the table. So, I added this. It doesn't occur on successive run when the test passes. I thought better to have this. I can try removing it, If you feel so ## hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java: ## @@ -26,8 +26,8 @@ import org.apache.hadoop.hbase.KeyValue; import org.apache.hadoop.hbase.KeyValueUtil; import org.apache.hadoop.hbase.client.Result; -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClientProtos; +import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos; Review Comment: The Hbase tests were failing because they had one protobuf 2.5.0 reference for some use case. So bugged the HBase folks and got that as a solution. https://issues.apache.org/jira/browse/HBASE-21587?focusedCommentId=17575116=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17575116 Issue Time Tracking --- Worklog Id: (was: 798408) Time Spent: 3h 40m (was: 3.5h) > Bump protobuf version to 3.21.1 (aarch64 support) > - > > Key: HIVE-24483 > URL: https://issues.apache.org/jira/browse/HIVE-24483 > Project: Hive > Issue Type: Improvement >Reporter: Cameron Moberg >Assignee: Ayush Saxena >Priority: Minor > Labels: pull-request-available > Time Spent: 3h 40m > Remaining Estimate: 0h > > The following protoc version's used in hive is very old i.e. 2.5.0 > [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The > v2.5.0 does not have aarch64 support. But the AArch64 support started from > v3.5.0 on-words in google's protobuf project release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[
https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=798403=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-798403
]
ASF GitHub Bot logged work on HIVE-24483:
-
Author: ASF GitHub Bot
Created on: 05/Aug/22 11:54
Start Date: 05/Aug/22 11:54
Worklog Time Spent: 10m
Work Description: zabetak commented on code in PR #3498:
URL: https://github.com/apache/hive/pull/3498#discussion_r938731653
##
standalone-metastore/metastore-common/pom.xml:
##
@@ -457,7 +457,7 @@
run
-
${protobuf.group}:protoc:${protobuf-exc.version}
+
com.google.protobuf:protoc:3.21.4
Review Comment:
Should we use `${protobuf.version}` instead of hardcoding `3.21.4`?
##
hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java:
##
@@ -26,8 +26,8 @@
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.KeyValueUtil;
import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
-import org.apache.hadoop.hbase.protobuf.generated.ClientProtos;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
Review Comment:
Do we know why these imports changed and point to shaded classes?
##
hbase-handler/src/java/org/apache/hadoop/hive/hbase/ResultWritable.java:
##
@@ -26,8 +26,8 @@
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.KeyValueUtil;
import org.apache.hadoop.hbase.client.Result;
-import org.apache.hadoop.hbase.protobuf.ProtobufUtil;
-import org.apache.hadoop.hbase.protobuf.generated.ClientProtos;
+import org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil;
+import org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos;
Review Comment:
Do we know why these imports changed and point to shaded classes?
##
data/files/datasets/src_hbase/load.hive.sql:
##
@@ -1,3 +1,5 @@
+DROP table if exists src_hbase_tmp;
Review Comment:
Seems reasonable but why we didn't do it before and we need to it now?
Issue Time Tracking
---
Worklog Id: (was: 798403)
Time Spent: 3.5h (was: 3h 20m)
> Bump protobuf version to 3.21.1 (aarch64 support)
> -
>
> Key: HIVE-24483
> URL: https://issues.apache.org/jira/browse/HIVE-24483
> Project: Hive
> Issue Type: Improvement
>Reporter: Cameron Moberg
>Assignee: Ayush Saxena
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> The following protoc version's used in hive is very old i.e. 2.5.0
> [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The
> v2.5.0 does not have aarch64 support. But the AArch64 support started from
> v3.5.0 on-words in google's protobuf project release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[ https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=798399=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-798399 ] ASF GitHub Bot logged work on HIVE-24483: - Author: ASF GitHub Bot Created on: 05/Aug/22 11:25 Start Date: 05/Aug/22 11:25 Worklog Time Spent: 10m Work Description: ayushtkn commented on PR #3498: URL: https://github.com/apache/hive/pull/3498#issuecomment-1206339881 Build is green. Tried compiling on OS-X-Aarch64 and I was able to get a successful `mvn clean install -DskipTests` without any hacks or tweaks. Awaiting reviews now. :-) Issue Time Tracking --- Worklog Id: (was: 798399) Time Spent: 3h 20m (was: 3h 10m) > Bump protobuf version to 3.21.1 (aarch64 support) > - > > Key: HIVE-24483 > URL: https://issues.apache.org/jira/browse/HIVE-24483 > Project: Hive > Issue Type: Improvement >Reporter: Cameron Moberg >Assignee: Ayush Saxena >Priority: Minor > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > The following protoc version's used in hive is very old i.e. 2.5.0 > [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The > v2.5.0 does not have aarch64 support. But the AArch64 support started from > v3.5.0 on-words in google's protobuf project release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[ https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=798034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-798034 ] ASF GitHub Bot logged work on HIVE-24483: - Author: ASF GitHub Bot Created on: 04/Aug/22 14:08 Start Date: 04/Aug/22 14:08 Worklog Time Spent: 10m Work Description: slachiewicz commented on PR #3498: URL: https://github.com/apache/hive/pull/3498#issuecomment-1205309912 You shout also regenerate code running with Protobuf profile see #3500 Issue Time Tracking --- Worklog Id: (was: 798034) Time Spent: 3h 10m (was: 3h) > Bump protobuf version to 3.21.1 (aarch64 support) > - > > Key: HIVE-24483 > URL: https://issues.apache.org/jira/browse/HIVE-24483 > Project: Hive > Issue Type: Improvement >Reporter: Cameron Moberg >Assignee: Ayush Saxena >Priority: Minor > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > The following protoc version's used in hive is very old i.e. 2.5.0 > [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The > v2.5.0 does not have aarch64 support. But the AArch64 support started from > v3.5.0 on-words in google's protobuf project release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[ https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=798031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-798031 ] ASF GitHub Bot logged work on HIVE-24483: - Author: ASF GitHub Bot Created on: 04/Aug/22 13:59 Start Date: 04/Aug/22 13:59 Worklog Time Spent: 10m Work Description: ayushtkn commented on PR #3498: URL: https://github.com/apache/hive/pull/3498#issuecomment-1205297929 Got a green build with 3.7.1 version(the red is due to oracle init failing), which is same as with one present in hadoop. Attempted to upgraded to 3.21.1 which is the latest. Tried building on AArch-64 as well and the build was successful Issue Time Tracking --- Worklog Id: (was: 798031) Time Spent: 3h (was: 2h 50m) > Bump protobuf version to 3.21.1 (aarch64 support) > - > > Key: HIVE-24483 > URL: https://issues.apache.org/jira/browse/HIVE-24483 > Project: Hive > Issue Type: Improvement >Reporter: Cameron Moberg >Assignee: Ayush Saxena >Priority: Minor > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > The following protoc version's used in hive is very old i.e. 2.5.0 > [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The > v2.5.0 does not have aarch64 support. But the AArch64 support started from > v3.5.0 on-words in google's protobuf project release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[ https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=797714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797714 ] ASF GitHub Bot logged work on HIVE-24483: - Author: ASF GitHub Bot Created on: 03/Aug/22 17:59 Start Date: 03/Aug/22 17:59 Worklog Time Spent: 10m Work Description: ayushtkn commented on PR #3498: URL: https://github.com/apache/hive/pull/3498#issuecomment-1204297458 yep, @Noremac201 I updated there and there were some explicit changes for Aarch because of lower version of protobuf, have updated that. Saw 2 Hbase related failure in the last build, may be upgrading Hbase version should solve them, will check post the next build result Issue Time Tracking --- Worklog Id: (was: 797714) Time Spent: 2h 50m (was: 2h 40m) > Bump protobuf version to 3.21.1 (aarch64 support) > - > > Key: HIVE-24483 > URL: https://issues.apache.org/jira/browse/HIVE-24483 > Project: Hive > Issue Type: Improvement >Reporter: Cameron Moberg >Assignee: Ayush Saxena >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > The following protoc version's used in hive is very old i.e. 2.5.0 > [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The > v2.5.0 does not have aarch64 support. But the AArch64 support started from > v3.5.0 on-words in google's protobuf project release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[ https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=797663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797663 ] ASF GitHub Bot logged work on HIVE-24483: - Author: ASF GitHub Bot Created on: 03/Aug/22 15:02 Start Date: 03/Aug/22 15:02 Worklog Time Spent: 10m Work Description: Noremac201 commented on PR #3498: URL: https://github.com/apache/hive/pull/3498#issuecomment-1204068244 Won't standalone-metastore/pom.xml also need to be updated? Issue Time Tracking --- Worklog Id: (was: 797663) Time Spent: 2h 40m (was: 2.5h) > Bump protobuf version to 3.21.1 (aarch64 support) > - > > Key: HIVE-24483 > URL: https://issues.apache.org/jira/browse/HIVE-24483 > Project: Hive > Issue Type: Improvement >Reporter: Cameron Moberg >Assignee: Ayush Saxena >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > The following protoc version's used in hive is very old i.e. 2.5.0 > [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The > v2.5.0 does not have aarch64 support. But the AArch64 support started from > v3.5.0 on-words in google's protobuf project release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-24483) Bump protobuf version to 3.21.1 (aarch64 support)
[ https://issues.apache.org/jira/browse/HIVE-24483?focusedWorklogId=797511=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-797511 ] ASF GitHub Bot logged work on HIVE-24483: - Author: ASF GitHub Bot Created on: 03/Aug/22 06:04 Start Date: 03/Aug/22 06:04 Worklog Time Spent: 10m Work Description: ayushtkn opened a new pull request, #3498: URL: https://github.com/apache/hive/pull/3498 Exploratory Sate: See what all breaks Issue Time Tracking --- Worklog Id: (was: 797511) Time Spent: 2.5h (was: 2h 20m) > Bump protobuf version to 3.21.1 (aarch64 support) > - > > Key: HIVE-24483 > URL: https://issues.apache.org/jira/browse/HIVE-24483 > Project: Hive > Issue Type: Improvement >Reporter: Cameron Moberg >Assignee: Ayush Saxena >Priority: Minor > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > The following protoc version's used in hive is very old i.e. 2.5.0 > [https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/] . The > v2.5.0 does not have aarch64 support. But the AArch64 support started from > v3.5.0 on-words in google's protobuf project release. -- This message was sent by Atlassian Jira (v8.20.10#820010)
