fter
> changing parameter
>
> spark.sql.autoBroadcastJoinThreshold to 10
>
>
> Caused by: java.lang.IllegalArgumentException: Can't zip RDDs with unequal
> numbers of partitions
> at
> org.apache.spark.rdd.ZippedPartitionsBaseRDD.
hu, Mar 17, 2016 at 10:03 AM, Jiří Syrový <syrovy.j...@gmail.com>
> wrote:
> > Hi,
> >
> > any idea what could be causing this issue? It started appearing after
> > changing parameter
> >
> > spark.sql.autoBroadcastJoinThreshold to 100000
> >
>
Hi,
any idea what could be causing this issue? It started appearing after
changing parameter
*spark.sql.autoBroadcastJoinThreshold to 10*
Caused by: java.lang.IllegalArgumentException: Can't zip RDDs with unequal
numbers of partitions
number, GraphX jobs will throw:
java.lang.IllegalArgumentException: Can't zip RDDs with unequal numbers of
partitions
So my quick fix is to repartition the EdgeRDD to exactly the number of
parallelism. But I think this would lead to much network communication.
So is there any other better
Hi All,
I met the titled error. This exception occured in line 223, as shown below:
212 // read files
213 val lines =
sc.textFile(path_edges).map(line=line.split(,)).map(line=((line(0),
line(1)), line(2).toDouble)).reduceByKey(_+
_).cache
214
215 val