JacksonKjar commented on PR #59215: URL: https://github.com/apache/spark/pull/59215#issuecomment-6159375139
Thanks for the review. Addressed in bd03fd255d4 … f9f1b9519e1. ### RDD name `createDataFrame(rdd, beanClass)` again builds a `LogicalRDD` named after the source RDD. The bean encoder is now only used to convert values. ### Equivalence for existing JavaBeans Checking this found one more difference in the previous revision: `@Nonnull` boxed properties became nullable, because a Dataset's schema follows the serializer. The schema now comes from the encoder, as before. #### Unchanged For both the `List` and `RDD`/`JavaRDD` overloads, the following behaviors match the original `beansToRows` path (tests also pass on the original implementation): - Schema: field names, types and nullability, including `@Nonnull` reference and boxed properties - Rows for primitive, boxed and `@Nonnull` properties, nested beans, `List`, `Map`, `String[]`, `int[]`, `byte[]`, `BigDecimal`, `java.sql.Date`/`Timestamp`, `LocalDate` and `Instant`, including nulls in nullable properties - Plan: `LocalRelation` for the `List` overload, and `LogicalRDD` named after the source RDD for the RDD overload - Beans with only getters (no setters or no-arg constructor) - Without ANSI mode, a `BigDecimal` that doesn't fit the decimal type becomes null #### Changed These tests fail on the original implementation: - Java record classes now work. Previously they failed with `ArrayIndexOutOfBoundsException` - Enum properties are stored as their names. Previously they failed with `INVALID_EXTERNAL_VALUE` - Nested beans inside arrays, `List`s and `Map`s now work ([SPARK-25654](https://issues.apache.org/jira/browse/SPARK-25654)). Previously they failed with `INVALID_EXTERNAL_VALUE` - A null element fails with `NOT_NULL_ASSERT_VIOLATION`. Previously `INTERNAL_ERROR` (`List` overload) or `NullPointerException` (RDD overload) - With ANSI mode on, a `BigDecimal` that doesn't fit the decimal type fails with `EXPRESSION_ENCODING_FAILED` caused by `NUMERIC_VALUE_OUT_OF_RANGE`, as `createDataset` already does. Previously `NUMERIC_VALUE_OUT_OF_RANGE` directly The tests are in `JavaDataFrameSuite` and `JavaRecordDatasetSuite`; existing users of these overloads (`JavaBeanDeserializationSuite` and the MLlib Java suites) also pass. ### Connect array components Added an `ArrowEncoderSuite` round trip of a record with `int[]`, `String[]` and record-array components, including null arrays and elements. I also added records and beans nested in each other, and a `ClientE2ETestSuite` test against a real Connect server. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
