Re: Can't zip RDDs with unequal numbers of partitions

2016-03-20 Thread Jakob Odersky
fter > changing parameter > > spark.sql.autoBroadcastJoinThreshold to 10 > > > Caused by: java.lang.IllegalArgumentException: Can't zip RDDs with unequal > numbers of partitions > at > org.apache.spark.rdd.ZippedPartitionsBaseRDD.

Re: Can't zip RDDs with unequal numbers of partitions

2016-03-19 Thread Jiří Syrový
hu, Mar 17, 2016 at 10:03 AM, Jiří Syrový <syrovy.j...@gmail.com> > wrote: > > Hi, > > > > any idea what could be causing this issue? It started appearing after > > changing parameter > > > > spark.sql.autoBroadcastJoinThreshold to 100000 > > >

Can't zip RDDs with unequal numbers of partitions

2016-03-18 Thread Jiří Syrový
Hi, any idea what could be causing this issue? It started appearing after changing parameter *spark.sql.autoBroadcastJoinThreshold to 10* Caused by: java.lang.IllegalArgumentException: Can't zip RDDs with unequal numbers of partitions

Re:[GraphX] Can't zip RDDs with unequal numbers of partitions

2014-08-07 Thread Bin
number, GraphX jobs will throw: java.lang.IllegalArgumentException: Can't zip RDDs with unequal numbers of partitions So my quick fix is to repartition the EdgeRDD to exactly the number of parallelism. But I think this would lead to much network communication. So is there any other better

Can't zip RDDs with unequal numbers of partitions

2014-08-05 Thread Bin
Hi All, I met the titled error. This exception occured in line 223, as shown below: 212 // read files 213 val lines = sc.textFile(path_edges).map(line=line.split(,)).map(line=((line(0), line(1)), line(2).toDouble)).reduceByKey(_+ _).cache 214 215 val