AngersZhuuuu commented on pull request #29692: URL: https://github.com/apache/spark/pull/29692#issuecomment-690873256
> > after stream side executed, we will get the raw count of each partition and judge if it's skewed seriously, if skewed seriously and volume is large, repartition stream side to make stream side RDD average. > > So you add a shuffle to the stream side to stop it before the join node and get statistics? Yes, stop before join and get count row of each stream side partitions. re-shuffle stream side then join. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
