Shuheng Ma created SPARK-60279:
----------------------------------
Summary: Spark Connect GroupedData.pivot rejects None, date and
Decimal pivot values that classic accepts
Key: SPARK-60279
URL: https://issues.apache.org/jira/browse/SPARK-60279
Project: Spark
Issue Type: Bug
Components: Connect, PySpark
Affects Versions: 4.2.0, 5.0.0
Reporter: Shuheng Ma
h3. Description
In Spark Connect, {{GroupedData.pivot(pivot_col, values)}} only accepts values
of type bool, float, int or str. Values that classic PySpark accepts, such as
{{None}}, {{datetime.date}}, {{datetime.datetime}} and {{decimal.Decimal}},
raise {{NOT_EXPECTED_TYPE}}. A common way to hit this is to collect the pivot
values with {{distinct()}}: if the column has a NULL, the list contains
{{None}}.
h3. Reproducer
{code:python}
# spark = SparkSession.builder.remote("local[2]").getOrCreate() for Connect,
.master("local[2]") for classic
import datetime, decimal
df = spark.createDataFrame(
[(1, "a", datetime.date(2020, 1, 1), decimal.Decimal("1.5"), 10),
(2, None, datetime.date(2020, 1, 2), decimal.Decimal("2.5"), 20)],
"k int, s string, d date, dec decimal(10,1), v int")
vals = [r[0] for r in df.select("s").distinct().collect()] # contains None
df.groupBy("k").pivot("s", vals).sum("v").columns
# classic: ['k', 'a', 'null']
# Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
bool, float, int or str, got NoneType.
df.groupBy("k").pivot("d", [datetime.date(2020, 1, 1)]).sum("v").columns
# classic: ['k', '2020-01-01']
# Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
bool, float, int or str, got date.
df.groupBy("k").pivot("dec", [decimal.Decimal("1.5")]).sum("v").columns
# classic: ['k', '1.5']
# Connect: PySparkTypeError: [NOT_EXPECTED_TYPE] Argument `value` should be
bool, float, int or str, got Decimal.
{code}
h3. Cause
{{GroupedData.pivot}} in {{python/pyspark/sql/connect/group.py}} checks each
value with {{isinstance(v, (bool, float, int, str))}}. The allowed types match
the {{LiteralType}} type hint rather than what {{F.lit}} supports. The values
are later turned into literals with {{F.lit}}, which handles None, date,
datetime and Decimal, so the check rejects values that would work. The check
has been there since the Connect pivot was added (SPARK-41693, 3.4.0).
Code:
https://github.com/apache/spark/blob/d18b51873dbcb8f18677b9a5860b3131f16b1d8e/python/pyspark/sql/connect/group.py#L248-L257
h3. Suggested fix
Also accept None, datetime.date (which covers datetime.datetime) and
decimal.Decimal in that check. Nested lists and dicts keep the current error;
classic rejects them too. Classic also accepts bytes, bytearray, datetime.time,
datetime.timedelta, NumPy scalars such as np.int64, np.bool_ and np.float32,
and a tuple of values; this fix does not cover them. Another option is to drop
the per-value check and let {{F.lit}} validate, as classic leaves it to the
JVM: that also covers bytes, time, timedelta and NumPy scalars, but a Column or
a nested list would then fail on the server with
{{CONNECT_INVALID_PLAN.UNSUPPORTED_LITERAL_TYPE}} instead of on the client.
h3. Who hits it
A feature pipeline on Databricks serverless (Spark Connect) hit the None case
and worked around it by dropping None from the values, which loses the "null"
column that classic would produce:
https://github.com/WenjieAtBayer/kerendia_nba_3_1_develop/pull/3
Tested on PySpark 4.2.0 from PyPI and on master (5.0.0-SNAPSHOT), Python 3.12,
with a local Spark Connect server (first line of the reproducer). I have a fix
with tests ready and will open a PR shortly.
Co-developed with Claude Code (Claude Opus 5.5): the investigation, the
reproducer and this description.
Generated-by: Claude Code (Claude Opus 5.5)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
