Yibo Dong created SPARK-59044:
---------------------------------
Summary: Excluding EliminateOffsets causes OFFSET 0 to fail during
physical planning
Key: SPARK-59044
URL: https://issues.apache.org/jira/browse/SPARK-59044
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Environment: Spark 5.0.0-SNAPSHOT, commit
9da9f8d673914d1648514f59d85e3adafb300d1a; macOS arm64; Java 17.0.17
Reporter: Yibo Dong
A legal OFFSET 0 query fails during physical planning when
EliminateOffsets is excluded.
h3. Reproduction
{code:sql}
SELECT 1 AS x OFFSET 0;
{code}
With the default optimizer configuration, the query executes successfully and
returns:
{code}
1
{code}
If only EliminateOffsets is excluded:
{code}
--conf
spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.EliminateOffsets
{code}
the same query fails during physical planning with:
{code}
org.apache.spark.SparkException: [INTERNAL_ERROR] The Spark SQL phase planning
failed with an internal error.
Caused by: java.lang.AssertionError: assertion failed
at org.apache.spark.sql.execution.CollectLimitExec.<init>(limit.scala:49)
{code}
h3. Expected behavior
OFFSET 0 is a no-op, so both configurations should execute successfully and
return the same result:
{code}
1
{code}
h3. Actual behavior
With EliminateOffsets enabled, the query returns 1.
When EliminateOffsets is excluded, the logical OFFSET 0 reaches physical
planning and causes an assertion failure while constructing CollectLimitExec.
h3. Additional information
The only configuration change needed to reproduce the failure is excluding
EliminateOffsets.
This suggests that physical planning assumes OFFSET 0 has already been removed
by EliminateOffsets, although the optimizer rule can currently be excluded.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
