[ https://issues.apache.org/jira/browse/HUDI-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17053902#comment-17053902 ]
Vinoth Chandar commented on HUDI-516: ------------------------------------- [~lamber-ken] this should be resolved now with your change.. correct? feel free to close this out if so > Avoid need to import spark-avro package when submitting Hudi job in spark > ------------------------------------------------------------------------- > > Key: HUDI-516 > URL: https://issues.apache.org/jira/browse/HUDI-516 > Project: Apache Hudi (incubating) > Issue Type: Improvement > Components: Usability > Reporter: Udit Mehrotra > Assignee: lamber-ken > Priority: Major > Fix For: 0.6.0 > > > We are in the process of migrating Hudi to *spark 2.4.4* and using > *spark-avro* instead of the deprecated *databricks-avro* here > [https://github.com/apache/incubator-hudi/pull/1005/] > After this change, users would be required to specifically download > spark-avro while start spark-shell using: > {code:java} > --packages org.apache.spark:spark-avro_2.11:2.4.4 > {code} > This is because we are not shading this now in *hudi-spark-bundle*. One > reason for not shading this is because we are not sure of the implications of > shading a spark dependency in a jar which is being submitted to spark. > [~vinoth] pointed out that a possible concern could be that we will always be > shading spark-avro 2.4.4 which can affect users using higher versions of > Spark. > This Jira is to come up with a way to solve this usability issue. > -- This message was sent by Atlassian Jira (v8.3.4#803005)