Hi, I am following the spark streaming stateful application example to write a stateful application and here is the critical line of code.
val keyStateStream = actRegBatchCountStream.updateStateByKey(update, new HashPartitioner(ssc.sparkContext.defaultParallelism), true, initKeyStateRDD) I noticed from log that spark by default is using sort partition for shuffle. So my questions are 1) So if I use HashPartitioner here, how the sort and hash co-exist/co-work to the shuffle related to updateStateByKey? 2) Is there an successful example of using other Partitioner in updateStateByKey? Thanks. Tian -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/updateStateByKey-and-Partitioner-tp24999.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org