[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17114970#comment-17114970 ] sivabalan narayanan commented on HUDI-259: -- [~Pratyaksh]: any progress on this. > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Assignee: Pratyaksh Sharma >Priority: Major > Labels: bug-bash-0.6.0 > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998780#comment-16998780 ] Vinoth Chandar commented on HUDI-259: - Can we do hadoop 3 i.e make the project compile and run with hadoop 3, without moving to hive 3? are hive 3 and hadoop 3 somehow tied? > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Assignee: Pratyaksh Sharma >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998480#comment-16998480 ] Wenning Ding commented on HUDI-259: --- Hey [~Pratyaksh], I am also working on hadoop 3 support for Hudi. After I using Hadoop 3.x and Hive 3.x. The unit tests for hudi-hive module fail when they trying to start hive metastore and hiveserver2, are you facing the same issue? > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Assignee: Pratyaksh Sharma >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998238#comment-16998238 ] Pratyaksh Sharma commented on HUDI-259: --- Yes, this way you can build your jars for deployment purpose. :) > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Assignee: Pratyaksh Sharma >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997935#comment-16997935 ] Yanjia Gary Li commented on HUDI-259: - I am already using Hadoop 3 with Spark 2.4. So far so good :P I built Hudi with *mvn clean install -DskipTests -DskipITs* **not an ideal way but didn't see any problem on the cluster yet. > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Assignee: Pratyaksh Sharma >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997899#comment-16997899 ] Vinoth Chandar commented on HUDI-259: - I believe we will get some eyes on this after the holidays :) > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Assignee: Pratyaksh Sharma >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995398#comment-16995398 ] Pratyaksh Sharma commented on HUDI-259: --- [~garyli1019] This is still under progress and the work is not yet complete. However, please let me know which modules are you facing issues, I can try to help. > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Assignee: Pratyaksh Sharma >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995153#comment-16995153 ] Yanjia Gary Li commented on HUDI-259: - Hello, I recently started using Hadoop 3 and Spark 2.4. [https://github.com/apache/incubator-hudi/commit/7bc08cbfdce337ad980bb544ec9fc3dbdf9c#diff-832156391e3edd5b0ceb86007ce6ae41] enable me to compile Hudi with Hadoop 3, but some tests are failed. > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Assignee: Pratyaksh Sharma >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955809#comment-16955809 ] Pratyaksh Sharma commented on HUDI-259: --- Hi [~vinoth], yeah I compared the poms, and there are significant changes. Okay, let me try doing this and get back to you. I checked the Jira for Hive 3.x (https://issues.apache.org/jira/browse/HUDI-6). Will be checking that too whenever I get time. > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16952071#comment-16952071 ] Vinoth Chandar commented on HUDI-259: - Hi [~Pratyaksh] please use master branch for these changes.. Our first apache release is imminent and there are tons of changes to pom since 0.4.7. Can we just keep the scope of this ticket to just Hadoop version? By that I mean, we may not actually bump the hadoop version on the pom, but - do a build with `*-Dhadoop.version=3.1.0*`, fix compilation errors and make code changes necessary (ultimately build should also pass with hadoop 2.x version currently in pom) - Take the build above and run it on the integration test environment and ensure it passes. Most of the cloud vendors still are on hadoop 2.x in a major way. we cannot drop support for that. On hive and spark - Hive 3.x is a major issue since it has backwards incompatible changes (phew!) There is a separate issue tracking that - Spark 2.4 is what we are planning to move to. udit is already driving that. Please let me know if this makes sense > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[
https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951692#comment-16951692
]
Pratyaksh Sharma commented on HUDI-259:
---
Hi [~vinoth], here are the pom changes in hoodie-0.4.7 ->
# pom.xml - hadoop version updated to 3.1.0, hive version updated to 3.1.0,
spark version updated to 2.3.2 and hbase version updated to 2.0.2
# Also since our production kafka cluster is by default ssl enabled, I had to
update spark-streaming-kafka artifact to spark-streaming-kafka-0-10_2.11. Also
one supporting dependency of {{spark-sql-kafka-0-10_2.11}} had to be included
so as to be able to rewrite KafkaOffsetGen.java class.
After a long time, now I can focus on fixing test cases again, so thought of
discussing the changes with you here as suggested by you. :)
> Hadoop 3 support for Hudi writing
> -
>
> Key: HUDI-259
> URL: https://issues.apache.org/jira/browse/HUDI-259
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: Usability
>Reporter: Vinoth Chandar
>Priority: Major
>
> Sample issues
>
> [https://github.com/apache/incubator-hudi/issues/735]
> [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568]
> [https://github.com/apache/incubator-hudi/issues/898]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933324#comment-16933324 ] Vinoth Chandar commented on HUDI-259: - [~Pratyaksh] awesome. if its a lot of changes to poms, can we first discuss them here, before you spend a lot of time on it? Not a lot of people outside of HDP have moved to Hadoop 3 yet. So we could also be cautious. Ultimately, ensuring hudi can keep workiing with 2.x is still the bread-and-butter for our users. > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933126#comment-16933126 ] Pratyaksh Sharma commented on HUDI-259: --- With Hadoop 3.1.0, few Hoodie Test classes are not compiling because either their dependent classes are not present, or their name/package has changed. I am working on fixing them. [~vinoth] > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-259) Hadoop 3 support for Hudi writing
[ https://issues.apache.org/jira/browse/HUDI-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932902#comment-16932902 ] Vinoth Chandar commented on HUDI-259: - Good first step would be ensuring Hudi can compile against all of 2.7, 2.8, 2.9, 3.0 .. > Hadoop 3 support for Hudi writing > - > > Key: HUDI-259 > URL: https://issues.apache.org/jira/browse/HUDI-259 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability >Reporter: Vinoth Chandar >Priority: Major > > Sample issues > > [https://github.com/apache/incubator-hudi/issues/735] > [https://github.com/apache/incubator-hudi/issues/877#issuecomment-528433568] > [https://github.com/apache/incubator-hudi/issues/898] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
