Thanks Miguel...will have a read. Thanks Jacek...that looks incredibly useful.
:) Subject: Re: Simulate serialization when running local From: [email protected] Date: Sun, 14 Aug 2016 21:07:41 -0700 CC: [email protected]; [email protected] To: [email protected] Hi Ashic, Absolutely. Serialization errors can be caught locally by taking a test driven approach. I have written a blog post about this as I believe it's important to develop spark applications this way. If you're interested you can find my post at https://medium.com/@therevoltingx/test-driven-development-w-apache-spark-746082b44941#.egnvmicyb Thanks Sent from my iPhone On Aug 14, 2016, at 6:07 PM, Jacek Laskowski <[email protected]> wrote: Hi Ashic, Yes, there is one - local-cluster[N, cores, memory] - that you can use for simulating a Spark cluster of [N, cores, memory] locally. https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L2478 Pozdrawiam, Jacek Laskowski ---- https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Wed, Aug 10, 2016 at 10:24 AM, Ashic Mahtab <[email protected]> wrote: Hi, Is there a way to simulate "networked" spark when running local (i.e. master=local[4])? Ideally, some setting that'll ensure any "Task not serializable" errors are caught during local testing? I seem to vaguely remember something, but am having trouble pinpointing it. Cheers, Ashic. --------------------------------------------------------------------- To unsubscribe e-mail: [email protected]
