rangareddy commented on issue #9574: URL: https://github.com/apache/hudi/issues/9574#issuecomment-4841602348
The intermittent `ClassNotFoundException: org.apache.hudi.common.model.DeleteRecord` is a classloader-visibility issue, not a missing jar or data corruption. It only surfaces when a task reads a MOR log file that contains a delete block — that path Kryo-deserializes a `DeleteRecord[]`, and in a Livy/Spark REPL session that deserialize resolves the class through Spark's `ExecutorClassLoader`, which intermittently fails to find a non-REPL class like `DeleteRecord` (hence the ~1-in-10 occurrence). Resolution: put the Hudi bundle on the executor **system** classpath, not just `--jars`. Set it on both driver and executors so the parent/system loader can always find the class: ```sh spark.driver.extraClassPath=/path/hudi-spark-bundle.jar spark.executor.extraClassPath=/path/hudi-spark-bundle.jar ``` With the bundle on the system classpath, the parent loader resolves `DeleteRecord` reliably even when the REPL class-server lookup races. Running the read as a compiled `spark-submit` job (which uses the application classloader instead of the REPL loader) also avoids it. Closing this as resolved — please feel free to reopen if the error persists after applying the classpath change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
