Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

shardulm94 closed issue #10225: byte and short types in spark no longer auto coerce to int32 URL: https://github.com/apache/iceberg/issues/10225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2088423502 Just wanted to make sure you were aware reproducing is pretty simple ``` Author: jkolash Date: Thu Apr 25 19:23:22 2024 -0400 Failing test for issue

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078535487 Ok this is reproducing via the github actions build on my public fork https://github.com/jkolash/iceberg/actions/runs/8842101257/job/24280206652 ``` TestDataFrameWriterV2 >

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078200948 hmm I think this may be related to the spark version we are using as I tested on spark-3.4.1 and didn't see the issue but see it on our 3.4.2 -- This is an automated message

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078157080 If/when there is a PR I can test it on my side. where I have exhaustive type testing. ``` java.lang.ClassCastException: class java.lang.Byte cannot be cast to class

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078150463 ```kotlin val df = spark.sql("""select inline(array(from_json('{"b":82}', 'struct')))""") df.show() ``` ``` +---+ | b| +---+ | 82|

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078099689 @Fokko Thanks for the quick response I will try to write up a code snippet reproducing the issue. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] byte and short types in spark no longer auto coerce to int32 [iceberg]

Fokko commented on issue #10225: URL: https://github.com/apache/iceberg/issues/10225#issuecomment-2078097905 Hey @jkolash Thanks for reporting this. The behavior should stay the same, due to the logic here:

[I] byte and short types in spark no longer auto coerce to int32 [iceberg]

jkolash opened a new issue, #10225: URL: https://github.com/apache/iceberg/issues/10225 ### Apache Iceberg version 1.5.0 ### Query engine Spark ### Please describe the bug  The removal of the code ```java private static PrimitiveWriter