Re: [Spark Core]: Python and Scala generate different DAGs for identical code

bugging I've got a YouTube video on the topic which >> could be a good intro (of course I'm pretty biased about that). >> >> On Wed, May 10, 2017 at 9:42 AM Pavel Klemenkov <pklemen...@gmail.com> >> wrote: >> >>> Thanks for the quick answer, Hold

Re: [Spark Core]: Python and Scala generate different DAGs for identical code

>> string is really useless for debugging. >> >> >> >> -- >> View this message in context: http://apache-spark-user-list. >> 1001560.n3.nabble.com/Spark-Core-Python-and-Scala- >> generate-different-DAGs-for-identical-code-tp28674.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> -- > Cell : 425-233-8271 <(425)%20233-8271> > Twitter: https://twitter.com/holdenkarau > -- Yours faithfully, Pavel Klemenkov.

[Spark Core]: Python and Scala generate different DAGs for identical code

java:0 [] | ../log.txt HadoopRDD[7] at textFile at NativeMethodAccessorImpl.java:0 [] Why is that? Does pyspark do some optimizations under the hood? This debug string is really useless for debugging. -- Yours faithfully, Pavel Klemenkov.