nevzheng opened a new issue, #12057: URL: https://github.com/apache/gravitino/issues/12057
### Describe the subtask Complete and document the five-family compatibility contract for Spark 3.3, 3.4, and 3.5. **Compatibility outcomes** - **Nanosecond timestamps:** Reject explicitly parameterized `timestamp(p)` and `timestamp_tz(p)` when `p > 6`, including precision 9. Preserve existing behavior for precision 6 and below. Reuse the work in #11941. - **Variant:** Reject top-level and nested Variant fields because Spark 3.x has no lossless equivalent. - **Unknown/Null:** Round-trip nullable Unknown only for Spark 3.4/3.5 Iceberg catalogs using Iceberg 1.11. Reject required fields, Spark 3.3, and non-Iceberg catalogs. - **Geometry:** Reject top-level and nested fields rather than dropping WKB or CRS semantics. - **Geography:** Reject top-level and nested fields rather than dropping WKB, CRS, or edge-algorithm semantics. **Acceptance** - Reproduce all five families through the connector's public create/conversion path and its alter/load paths where supported. - A supported mapping must round-trip without losing type identity or parameters. - An unsupported mapping must return `IllegalArgumentException` before external metadata mutation; no fallback to a narrower or opaque type is allowed. - Cover nested types where the connector supports complex columns. - Add focused converter/operation tests and an integration or no-side-effect test appropriate to the connector. - Update the connector documentation with a five-family compatibility matrix and rejection rationale. - Keep implementation commits atomic by connector and type family. **Connector-specific verification** Exercise the five families across supported Spark versions and catalog modes. Verify the supported Unknown mapping round-trips, every rejected case fails before table exposure or mutation, and precision 6-and-below timestamp behavior is unchanged. Run the shared Spark converter suite plus Spark 3.4 and 3.5 Iceberg converter suites. **Documentation** `docs/spark-connector/spark-catalog-iceberg.md` ### Parent issue #12056 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
