Hello Michael, Thanks for your quick reply.
My question wrt Java/Scala was related to extending the classes to support new custom data sources, so was wondering if those could be written in Java, since our company is a Java shop. The additional push downs I am looking for are aggregations with grouping and sorting. Essentially, I am trying to evaluate if this API can give me much of what is possible with the Apache MetaModel project. Regards, Ashish On Tue, Mar 24, 2015 at 1:57 PM, Michael Armbrust <mich...@databricks.com> wrote: > On Tue, Mar 24, 2015 at 12:57 AM, Ashish Mukherjee < > ashish.mukher...@gmail.com> wrote: >> >> 1. Is the Data Source API stable as of Spark 1.3.0? >> > > It is marked DeveloperApi, but in general we do not plan to change even > these APIs unless there is a very compelling reason to. > > >> 2. The Data Source API seems to be available only in Scala. Is there any >> plan to make it available for Java too? >> > > We tried to make all the suggested interfaces (other than CatalystScan > which exposes internals and is only for experimentation) usable from Java. > Is there something in particular you are having trouble with? > > >> 3. Are only filters and projections pushed down to the data source and >> all the data pulled into Spark for other processing? >> > > For now, this is all that is provided by the public stable API. We left a > hook for more powerful push downs > (sqlContext.experimental.extraStrategies), and would be interested in > feedback on other operations we should push down as we expand the API. >
