SEZ9 commented on issue #12753: URL: https://github.com/apache/seatunnel/issues/12753#issuecomment-6206430550
@zhengxiang378928908-code Sorry for the delayed answer to your direct question: I'm not picking either of the two options as written. Option 1 (an implicit `DecimalType(>38) -> StringType` fallback in the shared Spark `TypeConverterUtils`) is a cross-connector schema-contract change and shouldn't ride along with the Doris fix, and option 2 (disabling the affected cases on Spark containers) would hide a regression that today's `DecimalType(20,0)` mapping doesn't have. The failure you reproduced in https://github.com/zhengxiang378928908-code/seatunnel/actions/runs/38510021504 (`decimal can only support precision up to 38` at source schema translation, on Spark 2.4.6 and 3.3.0) needs its own explicit policy, which is exactly what the dedicated Spark compatibility issue you opened afterwards is for. So the path you landed on in your last comment is the right one, and thanks for also noting that PostgreSQL `NUMERIC(p > 38)` and the MySQL `DECIMAL(65)` clamp are already affected — tha t confirms the policy belongs at the Spark translation layer, not in the Doris connector. Where things stand from my side: - Agreed: the `DECIMAL(39, 0)` mapping for `LARGEINT` in `AbstractDorisTypeConverter`, with no clamping, is the source representation. Zeta and Flink passing on your fork is the coverage I expect the PR to carry. - Left in the PR: the `incompatible-changes.md` entry and the `ARRAY<largeint>` matching so `ARRAY<LARGEINT>` gets the same `DECIMAL(39, 0)` element type. Please ping me once both are pushed and I'll review. - Spark policy issue: it stays the blocker for merging the PR. Whichever of the three approaches you compared (documented string representation, preflight failure, or the schema-preserving `TIMESTAMP_TZ`-style metadata pattern) ends up preferred, please make sure the proposal covers the row write/read and nested-type paths, not only schema construction. If you already have a leaning among the three, say so there and I'll weigh in. <!-- streview-comment:1781 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
