subhramit opened a new pull request, #58225: URL: https://github.com/apache/spark/pull/58225
Closes [SPARK-58945](https://issues.apache.org/jira/browse/SPARK-58945) Note that the following description was initially generated using GPT 5.4, and then edited by me. ### What changes were proposed in this pull request? This fixes several broken Spark error-reporting paths caused by mismatches between `messageParameters` keys and the placeholders declared in [`error-conditions.json`](https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/resources/error/error-conditions.json). The changes fall into two groups: 1. **Fix Scala-side parameter key mismatches** - [`sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala`](https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala#L4678) - `_LEGACY_ERROR_TEMP_2450`: use `clazz` instead of `invalidClass` - [`sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala`](https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala) - `INVALID_PARAMETER_VALUE.EXTENSION`: use `invalidValue` instead of `fileExtension` / `acceptable` - `INVALID_WRITER_COMMIT_MESSAGE`: use `detail` instead of `details` - [`sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreErrors.scala`](https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreErrors.scala#L399-L403) - `STATE_STORE_COLUMN_FAMILY_SCHEMA_INCOMPATIBLE`: use `colFamilyName` / `oldSchema` / `newSchema` - [`sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala`](https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/sql/core/src/main/scala/org/apache/spark/sql/jdbc/H2Dialect.scala#L226-L232) - `TABLE_OR_VIEW_NOT_FOUND`: add the missing `searchPath` parameter as `"not available"`. `classifyException` receives only pre-rendered strings from `JDBCTableCatalog.scala:104-106`, so no Spark-side search path exists at that point. 2. **Fix one duplicated JSON message template** - [`common/utils/src/main/resources/error/error-conditions.json`](https://github.com/apache/spark/blob/fa6f71301587395d3576acb09b7bba2d8d3afc74/common/utils/src/main/resources/error/error-conditions.json#L11721) - `_LEGACY_ERROR_TEMP_3069` and `_LEGACY_ERROR_TEMP_3070` had byte-identical message templates. - `_LEGACY_ERROR_TEMP_3069` is the reserved-column-name collision case and its message is correct. - `_LEGACY_ERROR_TEMP_3070` is the unrecognized file metadata field fallback, and its Scala call site (`Map("field" -> field.toString)`) was already correct. This PR fixes only the template for `_LEGACY_ERROR_TEMP_3070`, restoring the pre-error class-migration wording: `Unrecognized file metadata field: <field>`. This is the text the branch threw before the error-class migration in [SPARK-46351](https://issues.apache.org/jira/browse/SPARK-46351), still visible at the `v3.5.0` tag. Also adds regression coverage for the fixed paths: - `sql/core/src/test/scala/org/apache/spark/sql/errors/ErrorMessageParametersSuite.scala` - `sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala` - `sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/jdbc/JDBCTableCatalogSuite.scala` - `sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceCustomMetadataStructSuite.scala` A minimal repro for one affected path is: ```scala spark.range(1).write.option("extension", "12").csv(path) ``` Before this change, that path fails during message rendering and surfaces `INTERNAL_ERROR` instead of `INVALID_PARAMETER_VALUE.EXTENSION`. ### Why are the changes needed? Spark's `StringSubstitutor` defaults to `enableUndefinedVariableException = true` (`common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala:29`), so an unresolved placeholder throws `IllegalArgumentException`, which `ErrorClassesJSONReader` converts into `SparkException.internalError`. Spark renders error messages eagerly when constructing many exceptions. The relevant exception constructors call `SparkThrowableHelper.getMessage` during construction (for example `common/utils/src/main/scala/org/apache/spark/SparkException.scala:288`). When substitution fails, the intended exception instance is never created, so the affected paths return `INTERNAL_ERROR` in place of the actual diagnosis. These are user-visible diagnostics bugs rather than behavior changes in the main execution path. Two of the six sites are not demonstrated by an end-to-end repro today. `stateStoreColumnFamilyMismatch` currently has no callers, so that path is latent. It would have failed during message construction before this fix, so the constructor is covered directly by a unit test. The `H2Dialect.scala:230` change is also covered directly rather than through `loadTable(...)`: missing-table `loadTable` is intercepted earlier by `JDBCRDD.resolveTable(...)`, so that path does not exercise the dialect branch. The dialect classification path itself is fixed here by supplying the missing `searchPath` parameter. For `_LEGACY_ERROR_TEMP_3070` the failure mode differs: the Scala call site was already correct, but the JSON template was duplicated from `_LEGACY_ERROR_TEMP_3069` during the error-class migration in [SPARK-46351](https://issues.apache.org/jira/browse/SPARK-46351), so the fallback reports a reserved-column-name collision that never occurred. ### Does this PR introduce _any_ user-facing change? Yes. This PR fixes user-facing error reporting for the currently reachable affected paths on master, and also fixes two latent error paths. Examples: - Invalid CSV writer `extension` values now report `INVALID_PARAMETER_VALUE.EXTENSION` instead of failing during message construction. - Invalid writer commit message counts now report `INVALID_WRITER_COMMIT_MESSAGE` with the intended detail text. - `_LEGACY_ERROR_TEMP_3070` now reports `Unrecognized file metadata field: <field>` instead of the unrelated reserved-column-name message. The H2 dialect change is a correctness fix in the JDBC classification path, but it is not demonstrated here by an end-to-end query repro. The state store fix is latent today because `stateStoreColumnFamilyMismatch` currently has no callers. This does not change the semantics of the underlying operations. It fixes the diagnostics that Spark surfaces when those error paths are hit. ### How was this patch tested? Added/updated regression tests: - `ErrorMessageParametersSuite` - `QueryExecutionErrorsSuite` - `JDBCTableCatalogSuite` - `FileSourceCustomMetadataStructSuite` Executed: ```bash build/sbt "core/testOnly *SparkThrowableSuite" build/sbt "sql/testOnly *ErrorMessageParametersSuite *QueryExecutionErrorsSuite" build/sbt "sql/testOnly *JDBCTableCatalogSuite *FileSourceCustomMetadataStructSuite" ./dev/scalastyle ./dev/run-tests ``` and all of them passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 + GPT-5.4 (Zed coding agent) + manually touched up in IntelliJ IDEA -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
