rangareddy commented on issue #17259: URL: https://github.com/apache/hudi/issues/17259#issuecomment-5508767396
Closing this as fixed. I ran the reproduction from the description against `master` at `a2788eef922e` and it no longer fails. ### Reproduction result Both inserts succeed and both rows read back correctly, where the second insert previously threw `SchemaParseException: Can't redefine: colName`: ``` REPRO_ROW_COUNT=2 REPRO_ROW=1.20,3.40 REPRO_ROW=5.60,7.80 Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0 ``` The table was created exactly as described, two structs whose nested member is named `colName` with different decimal precision, `decimal(11,2)` and `decimal(10,2)`, followed by the two inserts. ### Why it no longer collides The Avro `fixed` type backing a decimal is named after the field, so two same-named decimal members at the same namespace produced two `fixed` types called `colName` with different byte widths, which is what Avro rejects. `HoodieSparkSchemaConverters` now derives a per-level namespace when it descends into a struct, so the two members end up with distinct full names rather than both being `colName`: https://github.com/apache/hudi/blob/a2788eef922e2375432ead3fe50af7fe5f55faf7/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala#L199-L201 The decimal branch builds the `fixed` name from that namespace plus the field name, so `field1.colName` and `field2.colName` yield `...field1.colName.fixed` and `...field2.colName.fixed`: https://github.com/apache/hudi/blob/a2788eef922e2375432ead3fe50af7fe5f55faf7/hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/avro/HoodieSparkSchemaConverters.scala#L134-L140 ### Caveats worth recording No commit in the tree references HUDI-7602, so this was fixed incidentally rather than under this ticket. That means there is **no regression test pinning this specific shape** (two structs, same nested field name, differing decimal precision). If someone wants it locked down, a small SQL test along the lines of the description would be a cheap follow-up and is worth a separate ticket. I verified this on Spark 3.5 / Scala 2.12 only. The conversion path is shared across Spark versions, so I would not expect a divergence, but I did not run it on Spark 3.3, 3.4, or 4.x. The linked GitHub issue #10983 reports the same failure and can be closed alongside this one. HUDI-7602 in JIRA is still Open and will need resolving there as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
