nada-attia opened a new issue, #20762:
URL: https://github.com/apache/hudi/issues/20762

   ### Bug Description
   
   ### Summary:
   a MERGE INTO that updates an existing key can fail with HoodieIOException: 
Failed to parse record into provided schema (caused by EOFException or Invalid 
int encoding). In other cases it silently writes wrong, shifted values. It 
happens when the MERGE source columns are NOT NULL (for example a subquery of 
constants) while the table columns are nullable.
   
   ### Root cause
     - HoodieWriteMergeHandle.write builds the merge result and then calls 
recordContext.constructHoodieRecord(mergeResult).
     - For payload classes not on DEPRECATED_PAYLOADS, that wraps the merged 
row in a new payload. For MERGE INTO that is ExpressionPayload.
     - The merged row is already in the table schema, with nullable unions. 
shouldIgnore and writeUpdateRecord then call ExpressionPayload.getInsertValue, 
which decodes the row with the MERGE source schema 
(hoodie.payload.record.schema), not the table schema.
     - In BaseAvroPayload.getRecord, the schema-identity check fails. It 
re-serializes the record with the table schema, adding a 02 marker before each 
nullable value, then decodes those bytes with the non-null source schema. The 
shift gives an error or wrong values.
     - Built-in payloads aren't affected. They either get a plain 
HoodieAvroIndexedRecord from constructHoodieRecord already, or they decode with 
the schema passed in.
     - 0.14 isn't affected, because its merger returned an unwrapped record.
   
   **Steps to reproduce:**
   ```
     CREATE TABLE t (name STRING, city STRING, rating INT, ts BIGINT, country 
STRING)
     USING hudi PARTITIONED BY (country)
     TBLPROPERTIES (primaryKey = 'country,name', type = 'cow', preCombineField 
= 'ts');
   
     -- with 
hoodie.write.merge.handle.class=org.apache.hudi.io.HoodieWriteMergeHandle
     INSERT INTO t VALUES ('some name', 'phoenix', 1, 1000L, 'United_States');
   
   
     MERGE INTO t USING (
       SELECT 'some name' AS name, 'scottsdale' AS city, 9 AS rating, 2000L AS 
ts, 'United_States' AS country
     ) s
     ON t.country = s.country AND t.name = s.name
     WHEN MATCHED THEN UPDATE SET *
     WHEN NOT MATCHED THEN INSERT *;
   ```
     The same MERGE with a nullable source (a view whose columns are all 
nullable) works.
   
   ### Environment
   
   **Hudi version:** 1.2 
   **Query engine:** Spark
   **Relevant configs:** 
   ```
   hoodie.write.merge.handle.class=org.apache.hudi.io.HoodieWriteMergeHandle
   ```
   
   ### Logs and Stack Trace
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to