Daniel Fiterma created SPARK-41007: -------------------------------------- Summary: BigInteger Serialization doesn't work with JavaBean Encoder Key: SPARK-41007 URL: https://issues.apache.org/jira/browse/SPARK-41007 Project: Spark Issue Type: Bug Components: Java API Affects Versions: 3.3.1 Reporter: Daniel Fiterma
When creating a dataset using the [Java Bean Encoder|https://spark.apache.org/docs/3.2.0/api/java/org/apache/spark/sql/Encoders.html#bean-java.lang.Class-] with a bean that contains a field which is a {{java.math.BigInteger}} the dataset will fail to serialize correctly. When trying to deserialize the dataset, Spark throws the following error: {code:java} Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot up cast `bigInteger` from struct<> to decimal(38,18). {code} Reproduction steps: Using the Java Dataset API: # Create a Bean with a {{java.math.BigInteger}} field # Pass said Bean into the Java SparkSession {{createDataset}} function # -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org