[
https://issues.apache.org/jira/browse/SPARK-26979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-26979.
-------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 23882
[https://github.com/apache/spark/pull/23882]
> [PySpark] Some SQL functions do not take column names
> -----------------------------------------------------
>
> Key: SPARK-26979
> URL: https://issues.apache.org/jira/browse/SPARK-26979
> Project: Spark
> Issue Type: Improvement
> Components: PySpark
> Affects Versions: 2.4.0
> Reporter: Andre Sa de Mello
> Assignee: Andre Sa de Mello
> Priority: Minor
> Labels: easyfix, pull-request-available, usability
> Fix For: 3.0.0
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> Most SQL functions defined in _org.apache.spark.sql.functions_ have two
> variations, one taking a Column object as input, and another taking a string
> representing a column name, which is then converted into a Column object
> internally.
> There are, however, a few notable exceptions:
> * lower()
> * upper()
> * abs()
> * bitwiseNOT()
> While this doesn't break anything, as you can easily create a Column object
> yourself prior to passing it to one of these functions, it has two
> undesirable consequences:
> # It is surprising - it breaks coder's expectations when they are first
> starting with Spark. Every API should be as consistent as possible, so as to
> make the learning curve smoother and to reduce causes for human error;
> # It gets in the way of stylistic conventions. Most of the time it makes
> Python/Scala/Java code more readable to use literal names, and the API
> provides ample support for that, but these few exceptions prevent this
> pattern from being universally applicable.
> This is a very easy fix, and I see no reason not to apply it. I have a PR
> ready.
> *UPDATE:* Turns out there are many exceptions over this pattern that I wasn't
> aware of. The reason I missed them is because I had been looking at things
> from PySpark's point of view, and the API there does support column name
> literals for almost all SQL functions.
> Exceptions for the PySpark API include all the above plus:
> * ltrim()
> * rtrim()
> * trim()
> * ascii()
> * base64()
> * unbase64()
> The argument for making the API consistent still stands, however. I have been
> working on a PR to fix this on *PySpark's side*, and it should still be a
> painless change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
