prashantwason opened a new issue, #20528:
URL: https://github.com/apache/hudi/issues/20528

   ### Bug Description
   
   **What happened:**
   
   Found while checking a question from @yihua on #20483. A Hive/MR snapshot 
query on a COPY_ON_WRITE table fails when it reads a column whose Avro type is 
a union of more than one non-null type, such as `["null","int","string"]` or 
`["long", {"type":"record",...}]`. It fails as soon as a row holds a non-null 
value in that column:
   
   ```
   java.lang.IllegalArgumentException: should not be here?
        at 
org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchemaInternal(HoodieArrayWritableSchemaUtils.java:150)
        at 
org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchema(HoodieArrayWritableSchemaUtils.java:96)
        at 
org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteCanonicalRecord(HoodieArrayWritableSchemaUtils.java:184)
        at 
org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchemaInternal(HoodieArrayWritableSchemaUtils.java:110)
        at 
org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchema(HoodieArrayWritableSchemaUtils.java:96)
        at 
org.apache.hudi.hadoop.utils.HoodieArrayWritableSchemaUtils.rewriteRecordWithNewSchema(HoodieArrayWritableSchemaUtils.java:80)
        at 
org.apache.hudi.hadoop.HiveHoodieReaderContext.lambda$getFileRecordIterator$6(HiveHoodieReaderContext.java:258)
        at 
org.apache.hudi.common.util.collection.MappingIterator.next(MappingIterator.java:44)
        at 
org.apache.hudi.common.table.read.HoodieFileGroupReader.next(HoodieFileGroupReader.java:358)
        at 
org.apache.hudi.hadoop.HoodieFileGroupReaderBasedRecordReader.next(HoodieFileGroupReaderBasedRecordReader.java:194)
   ```
   
   The parquet writer stores such a union as a group of optional `member<N>` 
fields. 
[`HiveHoodieReaderContext#getFileRecordIterator`](https://github.com/apache/hudi/blob/8e5608ecaf6e3f78b33d83ac0bd4e4712cb81bd6/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/HiveHoodieReaderContext.java#L220-L258)
 reads the file with the file schema, where the column is that group (a 
record), and then rewrites each row to the required schema, which comes from 
the table schema, where the column is still a `UNION`. 
[`HoodieArrayWritableSchemaUtils#rewriteRecordWithNewSchemaInternal`](https://github.com/apache/hudi/blob/8e5608ecaf6e3f78b33d83ac0bd4e4712cb81bd6/hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/utils/HoodieArrayWritableSchemaUtils.java#L149-L150)
 has no handling for a `UNION` target and throws.
   
   The parquet read itself is fine: `setSchemas` builds `columns.types` from 
the file schema, so Hive's parquet reader gets `struct<member0:...>` and never 
sees `uniontype<...>`. With `hoodie.file.group.reader.enabled=false` the same 
queries return correct rows.
   
   **What you expected:**
   
   The query returns the union column as `struct<member0:...,memberN:...>`, the 
type that meta sync registers for it (#20483), with the branch that holds the 
value set and the others null. This is what the legacy reader already returns.
   
   **Steps to reproduce:**
   
   Reproduced with a unit test against `HoodieParquetInputFormat`:
   
   1. Create a COPY_ON_WRITE table and write one parquet base file with 
`AvroParquetWriter`, using this schema plus the meta fields:
      ```json
      {"type": "record", "name": "User", "fields": [
        {"name": "name", "type": "string"},
        {"name": "nullableunion", "type": ["null", "int", "string"], "default": 
null},
        {"name": "recordunion", "type": ["long", {"type": "record", "name": 
"Member", "fields": [
          {"name": "city", "type": "string"}, {"name": "count", "type": 
"int"}]}]},
        {"name": "unionarray", "type": {"type": "array", "items": ["int", 
"boolean"]}}]}
      ```
   2. Complete a commit whose metadata has that schema under `schema`.
   3. Set `columns` / `columns.types` to the types meta sync registers 
(`struct<member0:int,member1:string>`, 
`struct<member0:bigint,member1:struct<city:string,count:int>>`, 
`array<struct<member0:int,member1:boolean>>`). Then read the partition with 
`HoodieParquetInputFormat` and deserialize the rows with `ParquetHiveSerDe`.
   
   | Projection | File group reader (default) | 
`hoodie.file.group.reader.enabled=false` |
   |---|---|---|
   | `name` | ok | ok |
   | `nullableunion` | fails on the first non-null value | ok, e.g. 
`{"member0":1,"member1":null}` |
   | `recordunion, unionarray` | fails | ok |
   | all columns | fails | ok |
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT, 8e5608ecaf6e)
   **Query engine:** Hive (MR input format)
   **Relevant configs:** `hoodie.file.group.reader.enabled=true` (default)
   
   ### Logs and Stack Trace
   
   See above.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to