rangareddy commented on issue #16287: URL: https://github.com/apache/hudi/issues/16287#issuecomment-4891411543
This issue was reviewed as part of the JIRA-migrated backlog triage and reproduced/validated on the spark-hudi container (Spark 3.4) using the reporter's script (reconcile.schema=true + schema.on.read.enable=true). Findings: - The original NullPointerException (HUDI-7292) reproduces on 0.14.0/0.15.0 and is FIXED on 1.0.x/ 1.1.1 / 1.2.0 (no NPE). - However the same scenario still fails to READ on current master (1.2.0): count() throws java.lang.ClassCastException: org.apache.spark.sql.execution.vectorized.OnHeapColumnVector cannot be cast to ConstantColumnVector, from Spark's vectorized parquet reader (columnartorow_nextBatch in whole-stage codegen). - Isolated: the failure occurs only with the Spark vectorized parquet reader AND schema-on-read enabled. Reads succeed with either spark.sql.parquet.enableVectorizedReader=false OR hoodie.schema.on.read.enable=false. So the NPE is resolved, but a distinct read-path defect remains for schema-on-read tables under the vectorized reader. Keeping open (candidate to split into a new ticket) to fix the vectorized-reader / schema-on-read incompatibility. User workaround: set spark.sql.parquet.enableVectorizedReader=false when reading schema-on-read tables. Tracking as stream:fix / read_path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
