[GitHub] [hudi] codope commented on pull request #5674: [HUDI-4011] Add hudi-aws-bundle

[GitBox]

codope commented on PR #5674:
URL: https://github.com/apache/hudi/pull/5674#issuecomment-1143412485

   > LGTM. Better have some e2e test on this bundle jar alone to validate the 
functionality.
   
   Running this bundle jar with run_sync_tool was throwing below error
   ```
   Exception in thread "main" java.lang.NoSuchMethodError: 
org.apache.parquet.avro.AvroSchemaConverter.convert(Lorg/apache/parquet/schema/MessageType;)Lorg/apache/avro/Schema;
        at 
org.apache.hudi.common.table.TableSchemaResolver.convertParquetSchemaToAvro(TableSchemaResolver.java:351)
        at 
org.apache.hudi.common.table.TableSchemaResolver.getTableAvroSchemaFromDataFile(TableSchemaResolver.java:158)
        at 
org.apache.hudi.common.table.TableSchemaResolver.hasOperationField(TableSchemaResolver.java:575)
        at 
org.apache.hudi.common.table.TableSchemaResolver.<init>(TableSchemaResolver.java:83)
        at 
org.apache.hudi.sync.common.AbstractSyncHoodieClient.getDataSchema(AbstractSyncHoodieClient.java:164)
        at 
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:196)
        at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:142)
        at 
org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:130)
        at 
org.apache.hudi.aws.sync.AwsGlueCatalogSyncTool.main(AwsGlueCatalogSyncTool.java:68)
   ```
   Please check the last commit where I explicitly added parquet-avro 
dependencies and shaded from all other modules. After that I was able to run 
run_sync_tool with this bundle.
   
   <img width="1172" alt="Screenshot 2022-06-01 at 3 48 21 PM" 
src="https://user-images.githubusercontent.com/16440354/171382549-f876b8b6-75d3-41a0-bae3-29269869c902.png";>
   
   Note: These dependencies are not really required if we run it together with 
hudi-utilities-slim-bundle and hudi-spark-bundle. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org