Whatever benefits you may accrue from the rapid prototyping and coding in Python, it will be offset against the time taken to convert it to run inside the JVM. This of course depends on the complexity of the DAG. I guess it is a matter of language preference.
Regards, Sivakumaran S > On 02-Sep-2016, at 8:58 AM, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > > From an outsider point of view nobody likes change :) > > However, it appears to me that Scala is a rising star and if one learns it, > it is another iron in the fire so to speak. I believe as we progress in time > Spark is going to move away from Python. If you look at 2014 Databricks code > examples, they were mostly in Python. Now they are mostly in Scala for a > reason. > > HTH > > > > Dr Mich Talebzadeh > > LinkedIn > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw> > > http://talebzadehmich.wordpress.com <http://talebzadehmich.wordpress.com/> > > Disclaimer: Use it at your own risk. Any and all responsibility for any loss, > damage or destruction of data or any other property which may arise from > relying on this email's technical content is explicitly disclaimed. The > author will in no case be liable for any monetary damages arising from such > loss, damage or destruction. > > > On 2 September 2016 at 08:23, Jakob Odersky <ja...@odersky.com > <mailto:ja...@odersky.com>> wrote: > Forgot to answer your question about feature parity of Python w.r.t. Spark's > different components > I mostly work with scala so I can't say for sure but I think that all pre-2.0 > features (that's basically everything except Structured Streaming) are on > par. Structured Streaming is a pretty new feature and Python support is > currently not available. The API is not final however and I reckon that > Python support will arrive once it gets finalized, probably in the next > version. > >
