On Tue, Mar 24, 2015 at 12:57 AM, Ashish Mukherjee < ashish.mukher...@gmail.com> wrote: > > 1. Is the Data Source API stable as of Spark 1.3.0? >
It is marked DeveloperApi, but in general we do not plan to change even these APIs unless there is a very compelling reason to. > 2. The Data Source API seems to be available only in Scala. Is there any > plan to make it available for Java too? > We tried to make all the suggested interfaces (other than CatalystScan which exposes internals and is only for experimentation) usable from Java. Is there something in particular you are having trouble with? > 3. Are only filters and projections pushed down to the data source and > all the data pulled into Spark for other processing? > For now, this is all that is provided by the public stable API. We left a hook for more powerful push downs (sqlContext.experimental.extraStrategies), and would be interested in feedback on other operations we should push down as we expand the API.
