zhengxiang378928908-code commented on issue #12753: URL: https://github.com/apache/seatunnel/issues/12753#issuecomment-6182525569
@DanielLeens I implemented the `DECIMAL(39,0)` mapping as agreed, and before opening the PR I hit an engine-level limit that I think needs your call. **Spark can't represent `DECIMAL(39,0)`.** On my fork's CI run (https://github.com/zhengxiang378928908-code/seatunnel/actions/runs/38510021504), `doris-connector-it` passes on Zeta and every Flink container, but fails on Spark 2.4.6 and 3.3.0, on both JDK 11 and 17: ``` org.apache.spark.sql.AnalysisException: decimal can only support precision up to 38 at org.apache.spark.sql.types.DecimalType.<init>(DecimalType.scala:54) at org.apache.seatunnel.translation.spark.utils.TypeConverterUtils.convert(TypeConverterUtils.java:111) ... at org.apache.seatunnel.translation.spark.source.SeaTunnelSourceTable.schema(SeaTunnelSourceTable.java:100) ``` The failure happens while the Spark translation layer builds the source table schema, before any row is read. So with this change alone, **any Spark job that reads a Doris table with a LARGEINT column fails**, even when every value is small. Today the same job runs, and only values over 20 digits are a problem. On Spark this would be a regression, not a fix. The existing `DorisIT` and `DorisMultiReadIT` cases hit it too, not just the new Doris-to-MySQL E2E. This isn't a target-connector limit, so the "targets handle their own compatibility" split doesn't quite cover it. The two options I see: 1. **Handle it in the Spark translation layer** (`TypeConverterUtils`): map a SeaTunnel decimal with precision > 38 to Spark `StringType` and convert values accordingly, or fail with an explicit error that names the column. That touches shared code used by every connector, so I'd do it as a separate PR that lands first, then the LARGEINT fix on top. 2. **Keep the Doris fix as is and declare Spark unsupported for LARGEINT**: disable the affected cases on Spark containers with `@DisabledOnContainer` and document the limitation. Smaller, but it knowingly breaks LARGEINT reads on Spark that work today. I lean towards 1, since 2 trades a silent narrowing for a hard failure on Spark. I'd like your view before going either way. Clamping to 38 only on Spark is the other obvious workaround, but it goes against what you asked for here, so I've left it out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
