zhengruifeng opened a new pull request, #58468:
URL: https://github.com/apache/spark/pull/58468
### What changes were proposed in this pull request?
This PR aligns SQL function grouping across the Catalyst function registry,
the Scala API, and the
PySpark API reference.
It uses `ExpressionInfo.getGroup` as the canonical taxonomy because the
values are validated by
`ExpressionInfo.validGroups` and consumed by the SQL function documentation
generator. Specifically,
this PR:
* regroups all 594 Catalyst registry entries into helpers matching their
`ExpressionInfo` groups;
* aligns the Scala group ordering and moves `try_avg` and `try_sum` from
`math_funcs` to
`agg_funcs`;
* adds the missing `bitmap_and_agg` and `collect_union` entries to the
PySpark function reference;
and
* clarifies that physical section headings in the Scala and Python
implementation files are
navigation aids, while Scala `@group` tags and PySpark's grouped `__all__`
define their public API
grouping.
The registry remains split into helper methods to control bytecode size. The
largest helper after
regrouping is `aggregateExpressions`, with 88 entries and 2,485 bytes of
uninstrumented bytecode,
leaving substantial headroom for coverage instrumentation.
### Why are the changes needed?
The registry, Scala API, PySpark exports, and API documentation previously
used inconsistent group
organization. This made it difficult to determine where a function belongs
and easy for the
different surfaces to drift further apart. Aligning the group-defining
surfaces gives contributors
one consistent taxonomy without a large behavior-neutral reorder of
implementation methods.
### Does this PR introduce _any_ user-facing change?
No. The function implementations and runtime behavior are unchanged. The
Scala and PySpark API
documentation grouping becomes consistent with the SQL function metadata.
### How was this patch tested?
* `build/sbt 'sql-api/compile' 'catalyst/Test/compile'`
* `build/sbt 'catalyst/testOnly *LookupFunctionsSuite'`
* `build/sbt 'sql/testOnly *ExpressionInfoSuite'`
* `conda run -n spark-dev-313 python -m py_compile
python/pyspark/sql/functions/__init__.py
python/pyspark/sql/functions/builtin.py`
* Audited all 594 registry entries against their runtime `ExpressionInfo`
groups and compared the
overlapping Scala, PySpark export, and RST groups; no mismatches were
found.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
