Do we still have to keep the names of the functions distinct to avoid collisions in SQL? Or is there a plan to allow "importing" a namespace into SQL somehow?
I ask because if we have to keep worrying about name collisions then I'm not sure what the added complexity of #2 and #3 buys us. Punya On Wed, Apr 29, 2015 at 3:52 PM Reynold Xin <r...@databricks.com> wrote: > Scaladoc isn't much of a problem because scaladocs are grouped. Java/Python > is the main problem ... > > See > > https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions$ > > On Wed, Apr 29, 2015 at 3:38 PM, Shivaram Venkataraman < > shiva...@eecs.berkeley.edu> wrote: > > > My feeling is that we should have a handful of namespaces (say 4 or 5). > It > > becomes too cumbersome to import / remember more package names and having > > everything in one package makes it hard to read scaladoc etc. > > > > Thanks > > Shivaram > > > > On Wed, Apr 29, 2015 at 3:30 PM, Reynold Xin <r...@databricks.com> > wrote: > > > >> To add a little bit more context, some pros/cons I can think of are: > >> > >> Option 1: Very easy for users to find the function, since they are all > in > >> org.apache.spark.sql.functions. However, there will be quite a large > >> number > >> of them. > >> > >> Option 2: I can't tell why we would want this one over Option 3, since > it > >> has all the problems of Option 3, and not as nice of a hierarchy. > >> > >> Option 3: Opposite of Option 1. Each "package" or static class has a > small > >> number of functions that are relevant to each other, but for some > >> functions > >> it is unclear where they should go (e.g. should "min" go into basic or > >> math?) > >> > >> > >> > >> > >> On Wed, Apr 29, 2015 at 3:21 PM, Reynold Xin <r...@databricks.com> > wrote: > >> > >> > Before we make DataFrame non-alpha, it would be great to decide how we > >> > want to namespace all the functions. There are 3 alternatives: > >> > > >> > 1. Put all in org.apache.spark.sql.functions. This is how SQL does it, > >> > since SQL doesn't have namespaces. I estimate eventually we will have > ~ > >> 200 > >> > functions. > >> > > >> > 2. Have explicit namespaces, which is what master branch currently > looks > >> > like: > >> > > >> > - org.apache.spark.sql.functions > >> > - org.apache.spark.sql.mathfunctions > >> > - ... > >> > > >> > 3. Have explicit namespaces, but restructure them slightly so > everything > >> > is under functions. > >> > > >> > package object functions { > >> > > >> > // all the old functions here -- but deprecated so we keep source > >> > compatibility > >> > def ... > >> > } > >> > > >> > package org.apache.spark.sql.functions > >> > > >> > object mathFunc { > >> > ... > >> > } > >> > > >> > object basicFuncs { > >> > ... > >> > } > >> > > >> > > >> > > >> > > > > >
