You are only required to add classes to Kryo (compulsorily) if you use a
specific setting:
//require registration of all classes with Kyro
.set("spark.kryo.registrationRequired","true")
Here's an example of my setup, I think this is the best approach because
it forces me to really think
Thanks for your answer, you are correct, it's just a different approach
than the one I am asking for :)
Building an uber- or assembly- jar goes against the idea of placing the
jars on all workers. Uber-jars increase network traffic, using local:/
in the classpath reduces network traffic.
Hey Python 2.6 don't let the door hit you on the way out! haha Drop It
No Problem
On 01/05/2016 12:17 AM, Reynold Xin wrote:
Does anybody here care about us dropping support for Python 2.6 in
Spark 2.0?
Python 2.6 is ancient, and is pretty slow in many aspects (e.g. json
parsing) when
There is a lot of interesting info about this API here:
https://issues.apache.org/jira/browse/SPARK-5388
I got that from a comment thread on the last link in your PR. Thanks for
bringing this up! I knew you could check status via REST per
I am going to say no, but have not actually tested this. Just going on
this line in the docs:
http://spark.apache.org/docs/latest/configuration.html
|spark.driver.extraClassPath| (none) Extra classpath entries to
prepend to the classpath of the driver.
/Note:/ In client mode, this config
Hi, I got a good answer on the main question elsewhere, would anyone
please confirm the first code is the right approach? For a MVCE I am
trying to adapt this example and it's seems like I am having Java issues
with types:
(but this is basically the right approach?)
int count =
