Jubin Soni created SPARK-59909:
----------------------------------
Summary: PySpark raises internal errors instead of intended
messages due to undefined error conditions and mismatched message parameters
Key: SPARK-59909
URL: https://issues.apache.org/jira/browse/SPARK-59909
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.2.0, 4.0.0, 4.1.0
Reporter: Jubin Soni
Several PySpark call sites raise a PySparkException with an error condition
that does not exist in python/pyspark/errors/error-conditions.json, or with
messageParameters that do not match the condition's template. Constructing the
exception itself fails, so the user gets an unrelated internal error instead of
the intended message, often with a different exception type (e.g. ValueError
instead of TypeError).
h3. Undefined error conditions (fails with "ValueError: Cannot find main error
class ...")
* NOT_STR - DataStreamReader.changes() (pyspark/sql/streaming/readwriter.py).
NOT_STR was consolidated into NOT_EXPECTED_TYPE by SPARK-56179 while this new
usage was added concurrently.
* VARIANT_SIZE_LIMIT_EXCEEDED - pyspark/sql/variant_utils.py (3 places)
* INVALID_OPERATION_UUID_ID - pyspark/sql/connect/client/core.py
* UNSUPPORTED_FILTER - pyspark/sql/worker/data_source_pushdown_filters.py
h3. Mismatched message parameters (fails with "AssertionError: Undefined error
message parameter ...")
* DATA_SOURCE_EXTRANEOUS_FILTERS -
pyspark/sql/worker/data_source_pushdown_filters.py passes extra "input" and
"extraneous" parameters
* NO_ACTIVE_SESSION - pyspark/sql/connect/dataframe.py passes an extra
"operator" parameter
h3. Reproduction
{code:python}
>>> spark.readStream.changes(123)
ValueError: Cannot find main error class 'NOT_STR'
{code}
Expected: PySparkTypeError saying that `tableName` should be a str.
The existing test for DATA_SOURCE_EXTRANEOUS_FILTERS
(test_python_datasource.py) does not catch this, because it only asserts that
the condition name appears in the exception text, which is also true of the
AssertionError.
The JVM side guards against this in SparkThrowableSuite, but PySpark has no
equivalent check.
h3. Proposed fix
Fix each call site, and add a test that scans PySpark sources for errorClass
literals and verifies that each one is defined and that its literal
messageParameters keys match the template.
UNSUPPORTED_PIPELINES_DATASET_TYPE has the same kind of problem but is already
tracked in SPARK-57965, so it is out of scope here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
