Hello all. I have been running a Spark Job that eventually needs to do a large
join.
24 million x 150 million
A broadcast join is infeasible in this instance clearly, so I am instead
attempting to do it with Hash Partitioning by defining a custom partitioner as:
class RDD2Partitioner(partitions: Int) extends HashPartitioner(partitions) {
override def getPartition(key: Any): Int = key match {
case k: Tuple2[Int, String] => super.getPartition(k._1)
case _ => super.getPartition(key)
}
}
I then partition both arrays using this partitioner. However, the job
eventually fails with the following exception which if I had to guess indicated
that a network connection was interrupted during the shuffle stage, causing
things to get lost and ultimately resulting in a fetch failure:
14/11/14 12:56:21 INFO ConnectionManager: Removing ReceivingConnection to
ConnectionManagerId(innovationdatanode08.cof.ds.capitalone.com,37590)
14/11/14 12:56:21 INFO ConnectionManager: Key not valid ?
sun.nio.ch.SelectionKeyImpl@7369b398
14/11/14 12:56:21 INFO ConnectionManager: key already cancelled ?
sun.nio.ch.SelectionKeyImpl@7369b398
java.nio.channels.CancelledKeyException
at
org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:386)
at
org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
In the spark UI, I still see a substantial amount of shuffling going on at this
stage, I am wondering if I’m perhaps using the partitioner incorrectly?
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to
Capital One and/or its affiliates. The information transmitted herewith is
intended only for use by the individual or entity to which it is addressed. If
the reader of this message is not the intended recipient, you are hereby
notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error, please
contact the sender and delete the material from your computer.
