[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.0
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=612014&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-612014 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 18/Jun/21 21:09 Start Date: 18/Jun/21 21:09 Worklog Time Spent: 10m Work Description: belugabehr opened a new pull request #1742: URL: https://github.com/apache/hive/pull/1742 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 612014) Time Spent: 1h 40m (was: 1.5h) > Upgrade Hadoop to 3.3.0 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.0
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=576506&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-576506 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 04/Apr/21 00:20 Start Date: 04/Apr/21 00:20 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1742: URL: https://github.com/apache/hive/pull/1742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 576506) Time Spent: 1.5h (was: 1h 20m) > Upgrade Hadoop to 3.3.0 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.0
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=573103&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-573103 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 28/Mar/21 00:19 Start Date: 28/Mar/21 00:19 Worklog Time Spent: 10m Work Description: github-actions[bot] commented on pull request #1742: URL: https://github.com/apache/hive/pull/1742#issuecomment-808822021 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the d...@hive.apache.org list if the patch is in need of reviews. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 573103) Time Spent: 1h 20m (was: 1h 10m) > Upgrade Hadoop to 3.3.0 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.0
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=533651&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533651 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 10/Jan/21 04:05 Start Date: 10/Jan/21 04:05 Worklog Time Spent: 10m Work Description: vyaslav commented on pull request #1742: URL: https://github.com/apache/hive/pull/1742#issuecomment-756999283 > @wangyum Stuck. > > There are two big issues here: > > 1. Hive integration tests fire up Druid, Kafka, HDFS, LLAP, etc. all in the same JVM and their 3rd party dependencies are all over the place. Using a higher version of a dependency breaks one product, but using a lower version breaks the other. To make this work well, there probably needs to be a way to launch each service in their own JVM class loader. In lieu of that, I've been trying to move the ball closer to the goal post and getting dependencies closer together. > > [apache/druid#10683](https://github.com/apache/druid/pull/10683) > [HIVE-24542](https://issues.apache.org/jira/browse/HIVE-24542) > > 1. In HDFS 3.3.0, Hadoop team introduced `ProtobufRpcEngine2` in addition to `ProtobufRpcEngine` (sigh). Some of the Hive LLAP stuff is using this Hadoop Protobuf RPC engine (`ProtobufRpcEngine`). There's some `static` logic in the protocol engines that prohibits loading both RPC engines into the same JVM at the same time, I'm not sure why. HDFS was migrated to `ProtobufRpcEngine2`. So, again, in the integration tests, when the HDFS mini cluster is loaded, version 2 of the RPC engine is loaded into the JVM. When LLAP is later loaded, it fails to start because version 1 cannot be registered at the same time. Regarding 1st, I faced the same issues in my PR for upgrade to 3.1.3 - https://github.com/apache/hive/pull/1638 But, regarding 2nd I'am curious if it would be hard to replace `ProtobufRpcEngine` with `ProtobufRpcEngine2` in Hive itself. As I understand they have upgraded from PB2 to PB3 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 533651) Time Spent: 1h 10m (was: 1h) > Upgrade Hadoop to 3.3.0 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.0
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=533225&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-533225 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 08/Jan/21 21:09 Start Date: 08/Jan/21 21:09 Worklog Time Spent: 10m Work Description: vyaslav commented on pull request #1742: URL: https://github.com/apache/hive/pull/1742#issuecomment-756999283 > @wangyum Stuck. > > There are two big issues here: > > 1. Hive integration tests fire up Druid, Kafka, HDFS, LLAP, etc. all in the same JVM and their 3rd party dependencies are all over the place. Using a higher version of a dependency breaks one product, but using a lower version breaks the other. To make this work well, there probably needs to be a way to launch each service in their own JVM class loader. In lieu of that, I've been trying to move the ball closer to the goal post and getting dependencies closer together. > > [apache/druid#10683](https://github.com/apache/druid/pull/10683) > [HIVE-24542](https://issues.apache.org/jira/browse/HIVE-24542) > > 1. In HDFS 3.3.0, Hadoop team introduced `ProtobufRpcEngine2` in addition to `ProtobufRpcEngine` (sigh). Some of the Hive LLAP stuff is using this Hadoop Protobuf RPC engine (`ProtobufRpcEngine`). There's some `static` logic in the protocol engines that prohibits loading both RPC engines into the same JVM at the same time, I'm not sure why. HDFS was migrated to `ProtobufRpcEngine2`. So, again, in the integration tests, when the HDFS mini cluster is loaded, version 2 of the RPC engine is loaded into the JVM. When LLAP is later loaded, it fails to start because version 1 cannot be registered at the same time. Regarding 1st, I faced the same issues in my PR for upgrade to 3.1.3 - https://github.com/apache/hive/pull/1638 But, regarding 2nd I'am curious if it would be hard to replace `ProtobufRpcEngine` with `ProtobufRpcEngine2` in Hive itself. As I understand they have upgraded from PB2 to PB3 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 533225) Time Spent: 1h (was: 50m) > Upgrade Hadoop to 3.3.0 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.0
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=528819&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-528819 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 28/Dec/20 16:06 Start Date: 28/Dec/20 16:06 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #1742: URL: https://github.com/apache/hive/pull/1742#issuecomment-751765866 @wangyum Stuck. There are two big issues here: 1. Hive integration tests fire up Druid, Kafka, HDFS, LLAP, etc. all in the same JVM and their 3rd party dependencies are all over the place. Using a higher version of a dependency breaks one product, but using a lower version breaks the other. To make this work well, there probably needs to be a way to launch each service in their own JVM class loader. In lieu of that, I've been trying to move the ball closer to the goal post and getting dependencies closer together. https://github.com/apache/druid/pull/10683 HIVE-24542 2. In HDFS 3.3.0, Hadoop team introduced `ProtobufRpcEngine2` in addition to `ProtobufRpcEngine` (sigh). Some of the Hive LLAP stuff is using this Hadoop Protobuf RPC engine (`ProtobufRpcEngine`). There's some `static` logic in the protocol engines that prohibits loading both RPC engines into the same JVM at the same time, I'm not sure why. HDFS was migrated to `ProtobufRpcEngine2`. So, again, in the integration tests, when the HDFS mini cluster is loaded, version 2 of the RPC engine is loaded into the JVM. When LLAP is later loaded, it fails to start because version 1 cannot be registered at the same time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 528819) Time Spent: 50m (was: 40m) > Upgrade Hadoop to 3.3.0 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24484) Upgrade Hadoop to 3.3.0
[ https://issues.apache.org/jira/browse/HIVE-24484?focusedWorklogId=528272&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-528272 ] ASF GitHub Bot logged work on HIVE-24484: - Author: ASF GitHub Bot Created on: 25/Dec/20 00:46 Start Date: 25/Dec/20 00:46 Worklog Time Spent: 10m Work Description: wangyum commented on pull request #1742: URL: https://github.com/apache/hive/pull/1742#issuecomment-751136951 Any update? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 528272) Time Spent: 40m (was: 0.5h) > Upgrade Hadoop to 3.3.0 > --- > > Key: HIVE-24484 > URL: https://issues.apache.org/jira/browse/HIVE-24484 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
