On Mon, Nov 12, 2018 at 7:33 AM Joe <j...@net2020.org> wrote: > Hello, > I was reading Spark 2.4.0 release docs and I'd like to find out more > about barrier execution mode. > In particular I'd like to know what happens when number of partitions > exceeds number of nodes (which I think is allowed, Spark tuning doc > mentions that)? >
The barrier execution mode is different. It needs to run tasks for all partitions together. So when the number of partitions is greater than number of nodes, it will wait until more nodes are available and print warning messages. > Does Spark guarantee that all tasks process all partitions > simultaneously? They will start all together. We provide a barrier() <http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.BarrierTaskContext@getTaskInfos():Array[org.apache.spark.BarrierTaskInfo]> method in the task scope to help simple coordination among tasks. > If not then how does barrier mode handle partitions that > are waiting to be processed? > If there are partitions waiting to be processed then I don't think it's > possible to send all data from given stage to a DL process, even when > using barrier mode? > Thanks a lot, > > Joe > > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
