ssimeonov commented on issue #24232: [SPARK-27297] [SQL] Add higher order functions to scala API URL: https://github.com/apache/spark/pull/24232#issuecomment-502193579 It's extremely weird and inconsistent for there to be SparkSQL functions with no DSL equivalent. It forces companies such as Swoop to create our own (e.g., https://gist.github.com/ssimeonov/8d902d0dfda934a79c3a46ec7dc0523d) yet bear the uncertainty as to what OSS Spark does, which is not a great outcome for the ecosystem. It would have been much better if SparkSQL and DSL support had been launched jointly. The growing size of `functions` feels like a red herring, but, if it is a concern, what prevents us from putting various categories of functions in other "namespaces", the way `Dataset` does with statistics & missing value functionality? Either way, delaying a decision on this functionality, by which I don't mean this specific PR, does not help. /cc @rxin
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
