[GitHub] [hudi] a0x edited a comment on issue #4442: [SUPPORT] PySpark(3.1.2) with Hudi(0.10.0) failed when querying spark sql
a0x edited a comment on issue #4442: URL: https://github.com/apache/hudi/issues/4442#issuecomment-1005363447 Finally I fixed this problem by removing aws deps in `packing/hudi-spark-bundle/pom.xml` and recompiling it myself. ```xml ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] a0x edited a comment on issue #4442: [SUPPORT] PySpark(3.1.2) with Hudi(0.10.0) failed when querying spark sql
a0x edited a comment on issue #4442:
URL: https://github.com/apache/hudi/issues/4442#issuecomment-1004486507
> I have the same issue when running hudi on emr. This issue seems to have
the same root cause as in this one: #4474 . The solution is to shade and
relocate aws dependencies introduced in hudi-aws:
>
> > For our internal hudi version, we shade aws dependencies, you can add
new relocation and build a new bundle package:
> > For example, to shade aws dependencies in spark, add following codes in
**packaging/hudi-spark-bundle/pom.xml**
> > ```
> >
> >
> > com.amazonaws.
> >
${spark.bundle.spark.shade.prefix}com.amazonaws.
> >
> > ```
>
> @xushiyan should this relocation be added to the official hudi release to
avoid such conflicts?
@kazdy Thank you! This should work.
But shall we shade all aws deps in Spark? I'm worrying about the side
effict, but let me have a try before replying in #4474
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hudi] a0x edited a comment on issue #4442: [SUPPORT] PySpark(3.1.2) with Hudi(0.10.0) failed when querying spark sql
a0x edited a comment on issue #4442:
URL: https://github.com/apache/hudi/issues/4442#issuecomment-1004486507
> I have the same issue when running hudi on emr. This issue seems to have
the same root cause as in this one: #4474 . The solution is to shade and
relocate aws dependencies introduced in hudi-aws:
>
> > For our internal hudi version, we shade aws dependencies, you can add
new relocation and build a new bundle package:
> > For example, to shade aws dependencies in spark, add following codes in
**packaging/hudi-spark-bundle/pom.xml**
> > ```
> >
> >
> > com.amazonaws.
> >
${spark.bundle.spark.shade.prefix}com.amazonaws.
> >
> > ```
>
> @xushiyan should this relocation be added to the official hudi release to
avoid such conflicts?
@kazdy Thank you! This should work.
But shall we shade all aws deps in Spark? I'm worrying about the side
effict, but let me have a try before replying in that issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
