JacksonKjar opened a new pull request, #59215: URL: https://github.com/apache/spark/pull/59215
### What changes were proposed in this pull request? This PR adds encoder support for Java records ([SPARK-55396](https://issues.apache.org/jira/browse/SPARK-55396)), so a record can be used as a typed `Dataset` element in Classic and Spark Connect. - **API**: new `Encoders.record(Class)`. `Encoders.bean(Class)` also now detects records and returns the same encoder. - **Encoder**: new `AgnosticEncoders.JavaRecordEncoder`. Fields follow component declaration order and are read through the component accessors; values are decoded with the canonical constructor. Records are inferred in `JavaTypeInference` next to JavaBeans, so records and beans can nest in each other, and components support the same types and nullability rules as bean properties (including `javax.annotation.Nonnull`). - **Generic records**: a record whose type parameters are unbound (a top-level `Encoders.record(Box.class)`, or a raw `Box` component) is rejected, because the types are erased. A generic record used as a component is supported when the enclosing type binds its type arguments, e.g. a `Box<String>` field in a record. - **Classic**: serializer and deserializer support in `SerializerBuildHelper` and `DeserializerBuildHelper`. - **Spark Connect**: `ArrowSerializer` and `ArrowDeserializer` support for `JavaRecordEncoder`. Accessors and the canonical constructor are looked up from the record's declared component types rather than the field encoders' types, so generic components (whose accessors and constructor parameters are erased) work. - **`createDataFrame(data, beanClass)` / `createDataFrame(rdd, beanClass)` in Classic** now build the DataFrame with the bean encoder, as Spark Connect already does, and `SQLContext.beansToRows` is removed. Previously the schema came from the bean encoder but values were converted by `beansToRows`, a separate reflection-based converter that only understands JavaBean getters, so a record class failed with an `ArrayIndexOutOfBoundsException`. - **Errors**: new `NOT_A_RECORD_CLASS` and `GENERIC_RECORD_NOT_SUPPORTED` error conditions. - **Docs**: a record added to the Java Dataset creation example used by the SQL getting started guide, and updated Scaladoc for `Encoders` and `createDataFrame`. ### Why are the changes needed? Records are Java's standard immutable data carrier, but Spark has no encoder for them. `Encoders.bean` requires JavaBean conventions (getters, setters and a no-arg constructor), which records don't follow, so a record passed to it produces an empty schema and fails when decoded. See also [this StackOverflow question](https://stackoverflow.com/questions/76092020/is-spark-3-3-2-that-can-run-under-java-17-able-to-manage-records). ### Does this PR introduce _any_ user-facing change? Yes. Java records can now be used as `Dataset` types in Classic and Spark Connect. `Encoders.record(Employee.class)`, `Encoders.bean(Employee.class)`, and `createDataFrame(list, Employee.class)` all use the same encoder, producing Datasets with one column per record component in declared order. For example: ```java public record Employee(String name, long salary) {} Dataset<Employee> ds = spark.createDataset( List.of(new Employee("Andy", 4000L)), Encoders.record(Employee.class)); ds.show(); // +----+------+ // |name|salary| // +----+------+ // |Andy| 4000| // +----+------+ ``` Previously (in 4.2.0 and on master), there was no `Encoders.record`, and records didn't work with the bean APIs: - `Encoders.bean(Employee.class)` produced an encoder with an empty schema, so `toDF()` returned rows with no columns and `collect()` failed with `EXPRESSION_DECODING_FAILED`. - `createDataFrame(list, Employee.class)` returned rows with no columns. Migrating the existing `createDataFrame(..., beanClass)` API to encoders introduces the following behavior changes: - Enum properties are now supported. Previously they failed with `INVALID_EXTERNAL_VALUE`. - Nested beans inside arrays, `List`s and `Map`s are now supported ([SPARK-25654](https://issues.apache.org/jira/browse/SPARK-25654)). Previously they failed with `INVALID_EXTERNAL_VALUE`. - With ANSI mode on, a `BigDecimal` that doesn't fit the decimal type now fails with `EXPRESSION_ENCODING_FAILED` caused by `NUMERIC_VALUE_OUT_OF_RANGE`, as `createDataset` already does, instead of `NUMERIC_VALUE_OUT_OF_RANGE` directly. Without ANSI mode it still becomes null. - A null element now fails with `NOT_NULL_ASSERT_VIOLATION` instead of an `INTERNAL_ERROR`. - The RDD overload's plan is `SerializeFromObject` over `ExternalRDD`, as for `createDataset(rdd)`, instead of `LogicalRDD`. - Schemas and results are otherwise unchanged; beans with only getters (no setters or no-arg constructor) keep working. ### How was this patch tested? New tests: - `JavaTypeInferenceSuite`: record encoder inference (field order, accessors, leaf, collection and array components, `@Nonnull`, nesting with beans, bound generic components) and error cases. - `JavaRecordEncoderSuite`: serializer/deserializer round trips, each run in both codegen and interpreted mode, and Java serialization of record encoders. - `JavaRecordDatasetSuite`: typed Dataset operations, records as grouping keys and in tuples, `as[Record]` by column name, null in a primitive component, SQL queries, and `createDataFrame` with a record class. - `ArrowEncoderSuite`: Spark Connect Arrow round trips of records with nested record, `List` and `Map` components, and generic record components (unbounded, bounded and interface-bounded type parameters). Existing suites run locally: `ExpressionEncoderSuite`, `JavaDatasetSuite`, `JavaDataFrameSuite`, `JavaBeanDeserializationSuite`, `SparkThrowableSuite`, the full `ArrowEncoderSuite`, and the MLlib Java suites that build DataFrames with `createDataFrame(..., Bean.class)`. ```bash build/sbt "catalyst/testOnly *JavaTypeInferenceSuite *JavaRecordEncoderSuite *ExpressionEncoderSuite" \ "sql/testOnly *JavaRecordDatasetSuite *JavaDatasetSuite *JavaDataFrameSuite *JavaBeanDeserializationSuite" \ "connect-client-jvm/testOnly *ArrowEncoderSuite" \ "core/testOnly *SparkThrowableSuite" ``` Interpreted (non-codegen) deserialization of a constructor that takes an object array fails for case classes and JavaBeans as well as records, so the record array round trip runs with codegen only. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Kiro CLI 2.27.0 (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
