Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan closed issue #9827: MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider URL: https://github.com/apache/hudi/issues/9827 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1757888560 Thank you for helping me. The problem has been resolved, so I will close this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1757878099 > m1 chip does not have good compatibility, can you try arm64 chip instead? thank you，give more suggestions. M1 chip is based on the arm64 architecture. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1757864136 I analyzed the reason for this issue based on the source code，sharing here hopes to be helpful to others . In master branch,when you create docker contain.It copy docker/demo and docker/hoodie/hadoop/hive_base/target/hoodie-utilities.jar into directory /var/hoodie/ws/docker/.. in adhoc-2 contain.So execute spark-submit, it use /var/hoodie/ws/docker/hoodie/hadoop/hive_base/target/hoodie-utilities.jar. But the jar may compiled by me in branch 13.0 code.So it should be caused by a version mismatch, because I did not compile the master branch code locally successfully, and my local hoodie-utilities.jar was previously compiled using branch 13.0. https://github.com/apache/hudi/assets/19281198/12a3e983-344d-44bb-967e-18b1d270660c";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1757792367 > @hanrongMan I was able to run the complete docker demo with 0.14.0 on M1 and didn't faced any issues. What build command you are using? As default profiles have changes, can you try with this - > > mvn -U clean package -Pintegration-tests -DskipTests -Dscala-2.11 -Dspark2.4 thank you . I switched to branch 14.0， compiled using your command, and run successfully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1757789513 > @hanrongMan I was able to run the complete docker demo with 0.14.0 on M1 and didn't faced any issues. What build command you are using? As default profiles have changes, can you try with this - > > mvn -U clean package -Pintegration-tests -DskipTests -Dscala-2.11 -Dspark2.4 thank you . I switched to branch 14.0， compiled using your command, and run successfully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1757781453 > m1 chip does not have good compatibility, can you try arm64 chip instead? apple M1 chip -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan closed issue #9827: MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider URL: https://github.com/apache/hudi/issues/9827 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
ad1happy2go commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1756058793 @hanrongMan I was able to run the complete docker demo with 0.14.0 on M1 and didn't faced any issues. What build command you are using? As default profiles have changes, can you try with this - mvn -U clean package -Pintegration-tests -DskipTests -Dscala-2.11 -Dspark2.4 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
danny0405 commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1752282559 m1 chip does not have good compatibility, can you try arm64 chip instead? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1751610045 > What I did according to this article ：https://hudi.apache.org/cn/docs/0.13.0/docker_demo#testing-hudi-in-local-docker-environment What I did according to this article ：https://hudi.apache.org/cn/docs/0.13.0/docker_demo#testing-hudi-in-local-docker-environment, I saw the logs "Required property hoodie.deltastreamer.schemaprovider.source.schema.file is missing", it seems jar downloaded caused. I would like to confirm that according to the introduction in this document, the docker running on the m1 chip must be the master branch. Do you need to compile it first? Is there anything missing from my previous steps -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
danny0405 commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1751608503 Maybe you can analyze you context why the schema is not set up correctly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1751564997 > `hoodie.deltastreamer.schemaprovider.source.schema.file` did you create your table with SQL ddl, did you set up any schema params from cmd line? I don‘t create the table, don't set up any schema params. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
danny0405 commented on issue #9827: URL: https://github.com/apache/hudi/issues/9827#issuecomment-1751562256 `hoodie.deltastreamer.schemaprovider.source.schema.file` did you create your table with SQL ddl, did you set up any schema params from cmd line? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] MacOs M1 Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider [hudi]
hanrongMan opened a new issue, #9827: URL: https://github.com/apache/hudi/issues/9827 OS : MacOS 14.0 (23A344) Chip : Apple M1 Docker Env: colima version 0.5.5 + Docker version 20.10.17, build 100c70180f All my operating steps are as follows： What I did according to this article ：https://hudi.apache.org/cn/docs/0.13.0/docker_demo#testing-hudi-in-local-docker-environment 1. I pulled the master branch，then cd docker directory； 2. I execute command： ./setup_demo.sh --mac-aarch64，the result logs shows： ``` Copying spark default config and setting up configs 23/10/06 14:00:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 23/10/06 14:00:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 23/10/06 14:00:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable ``` but ervery docker contain is ruuning normally. 3.I continue command `cat docker/demo/data/batch_1.json | kcat -b kafkabroker -t stock_ticks -P ` `kcat -b kafkabroker -L -J | jq .` The displayed results are consistent with those in the guidance document. 4. I continue command ` docker exec -it adhoc-2 /bin/bash` 5. I continue command ` spark-submit \ --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer $HUDI_UTILITIES_BUNDLE \ --table-type COPY_ON_WRITE \ --source-class org.apache.hudi.utilities.sources.JsonKafkaSource \ --source-ordering-field ts \ --target-base-path /user/hive/warehouse/stock_ticks_cow \ --target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties \ --schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider` error happens: here are logs: `23/10/06 14:02:13 INFO spark.SparkContext: Successfully stopped SparkContext Exception in thread "main" java.io.IOException: Could not load schema provider class org.apache.hudi.utilities.schema.FilebasedSchemaProvider at org.apache.hudi.utilities.UtilHelpers.createSchemaProvider(UtilHelpers.java:168) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.(HoodieDeltaStreamer.java:678) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.(HoodieDeltaStreamer.java:148) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.(HoodieDeltaStreamer.java:121) at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:573) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.utilities.schema.FilebasedSchemaProvider at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:79) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:106) at org.apache.hudi.utilities.UtilHelpers.createSchemaProvider(UtilHelpers.java:166) ... 16 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:77) ... 18 more Caused by: org.apache.hudi.exception.HoodieNotSupportedException: Required property hoodie.deltastreamer.schemaprovider.source.schema.file is missing at org.apache.hudi
