Re: Kotlin Spark API

2020-07-14 Thread Pasha Finkelshteyn
Hi Stephen, Thank you so much for finding time for looking at our examples! Yes, we've tried to implement as clean design of API as possible and are constantly looking for ways to make it even more readable, clear and friendly. And as Maria already stated we welcome any feedback! On 20/07/14

Re: Kotlin Spark API

2020-07-14 Thread Stephen Boesch
I just looked at the examples. https://github.com/JetBrains/kotlin-spark-api/tree/master/examples/src/main/kotlin/org/jetbrains/spark/api/examples These look v nice! V concise yet flexible. I like the ability to do inline *side-effects. *E.g. caching or printing or showDs() package

Re: Kotlin Spark API

2020-07-14 Thread Stephen Boesch
I started with scala/spark in 2012 and scala has been my go-to language for six years. But I heartily applaud this direction. Kotlin is more like a simplified Scala - with the benefits that brings - than a simplified java. I particularly like the simplified / streamlined collections classes.

Re: Kotlin Spark API

2020-07-14 Thread Anwar AliKhan
Is kotlin another new language ? GRADY BOOCH; The United States Department of defence (DOD) is perhaps the largest user of computers in the world. By the mid-1970s, software development for its systems had reached crisis proportions: projects were often late, over budget and they often failed to

Kotlin Spark API

2020-07-14 Thread Maria Khalusova
Hi folks, We would love your feedback on the new Kotlin Spark API that we are working on: https://github.com/JetBrains/kotlin-spark-api. Why Kotlin Spark API? Kotlin developers can already use Kotlin with the existing Apache Spark Java API, however they cannot take full advantage of Kotlin

Mock spark reads and writes

2020-07-14 Thread Dark Crusader
Sorry I wasn't very clear in my last email. I have a function like this: def main( read_file): df = spark.read.csv(read_file) ** Some other code ** df.write.csv(path) Which I need to write a unit test for. Would pythons unittest mock help me here? When I googled this, I

Re: Issue in parallelization of CNN model using spark

2020-07-14 Thread Anwar AliKhan
Ok, thanks. You can buy it here https://www.amazon.com/s?k=hands+on+machine+learning+with+scikit-learn+and+tensorflow+2=2U0P9XVIJ790T=Hands+on+machine+%2Caps%2C246=nb_sb_ss_i_1_17 This book is like an accompaniment to the Andrew Ng course on coursera. It uses exact same mathematical notations ,

Re: Issue in parallelization of CNN model using spark

2020-07-14 Thread Sean Owen
It is still copyrighted material, no matter its state of editing. Yes, you should not be sharing this on the internet. On Tue, Jul 14, 2020 at 9:46 AM Anwar AliKhan wrote: > > Please note It is freely available because it is an early unedited raw > edition. > It is not 100% complete , it is not

Re: Issue in parallelization of CNN model using spark

2020-07-14 Thread Anwar AliKhan
Please note It is freely available because it is an early unedited raw edition. It is not 100% complete , it is not entirely same as yours. So it is not piracy. I agree it is a good book. On Tue, 14 Jul 2020, 14:30 Patrick McCarthy, wrote: > Please don't advocate for piracy, this book is

Re: Spark Compatibility with Java 11

2020-07-14 Thread Ankur Mittal
Thanks a lot. On Tue, Jul 14, 2020 at 12:51 PM Prashant Sharma wrote: > Hi Ankur, > > Java 11 support was added in Spark 3.0. > https://issues.apache.org/jira/browse/SPARK-24417 > > Thanks, > > > On Tue, Jul 14, 2020 at 6:12 PM Ankur Mittal > wrote: > >> Hi, >> >> I am using Spark 2.X and need

Re: Issue in parallelization of CNN model using spark

2020-07-14 Thread Patrick McCarthy
Please don't advocate for piracy, this book is not freely available. I own it and it's wonderful, Mr. GĂ©ron deserves to benefit from it. On Mon, Jul 13, 2020 at 9:59 PM Anwar AliKhan wrote: > link to a free book which may be useful. > > Hands-On Machine Learning with Scikit-Learn, Keras, and

Re: Spark Compatibility with Java 11

2020-07-14 Thread Prashant Sharma
Hi Ankur, Java 11 support was added in Spark 3.0. https://issues.apache.org/jira/browse/SPARK-24417 Thanks, On Tue, Jul 14, 2020 at 6:12 PM Ankur Mittal wrote: > Hi, > > I am using Spark 2.X and need to execute Java 11 .Its not able to execute > Java 11 using Spark 2.X. > > Is there any way

Spark Compatibility with Java 11

2020-07-14 Thread Ankur Mittal
Hi, I am using Spark 2.X and need to execute Java 11 .Its not able to execute Java 11 using Spark 2.X. Is there any way we can use Java 11 with Spark2.X? Has this issue been resolved in Spark 3.0 ? -- Regards Ankur Mittal

Re:org.apache.spark.deploy.yarn.ExecutorLauncher not found when running Spark 3.0 on Hadoop

2020-07-14 Thread godlumen
If you are sure that there are yarn related jars in the jars directory, try to use --conf spark.yarn.jars=hdfs://namenode:8020/spark-3/jars/* -- Lumen At 2020-07-14 04:31:38, "ArtemisDev" wrote: I've been trying to set up the latest stable version of Spark 3.0 on a hadoop