linliu-code opened a new issue, #18382:
URL: https://github.com/apache/hudi/issues/18382
### Bug Description
**What happened:**
When the parquet files in the source folder have different schemas, e.g.,
file1.parquet contains columns c1, c2, file2.parquet contains columns c1, c3.,
without enforcing merging schema during bootstrap phase, Spark could use the
schema from either file1 or file2 as the final schema.
This behavior could cause either c2, or c3 column to be NULL for all records
ingested during the bootstrap.
**What you expected:**
The schema should be created by merging schemas from all parquet files
ingested in the bootstrap phase.
**Steps to reproduce:**
1. have two parquet files that have different schemas in the source folder
2. run the s3 incremental stream to ingest these files.
3.
### Environment
**Hudi version:** All versions
**Query engine:** (Spark/Flink/Trino etc): Spark
**Relevant configs:**
spark.read.format().option("mergeSchema", "true")....
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
