bvaradar commented on issue #625: fix(spark): fix shading for spark bundle to 
include all hive's libraries unshaded
URL: https://github.com/apache/incubator-hudi/pull/625#issuecomment-481032184
 
 
   @lyogev : Thanks a lot for reporting the issue. We needed to selectively 
include and shade a subset of hive dependencies for hoodie to work across hive 
versions with Spark 
(http://hudi.incubator.apache.org/quickstart.html#version-compatibility). 
Having said that, there is an issue causing hive dependencies to not get 
included in hoodie-spark bundle. 
   
   Can you try this patch and see if this resolves the problem.
   
   https://gist.github.com/bvaradar/289134ac30f5103696918713e0d9bb1a
   
   As you expect, the plan is to use only one hudi bundle. You should include 
only hoodie-spark bundle if you are using spark-shell/spark-submit. If you are 
planning to use delta-streamer for ingesting data, you should only use 
hoodie-utilities bundle. This jar also has hive/spark dependencies included.
   
   Balaji.V
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to