Re: Does MapWithState follow with a shuffle ?

2016-11-29 Thread Shixiong(Ryan) Zhu
Right. And you can specify the partitioner via "StateSpec.partitioner(partitioner: Partitioner)". On Tue, Nov 29, 2016 at 1:16 PM, Amit Sela wrote: > Hi all, > > I've been digging into MapWithState code (branch 1.6), and I came across > the compute >

Does MapWithState follow with a shuffle ?

2016-11-29 Thread Amit Sela
Hi all, I've been digging into MapWithState code (branch 1.6), and I came across the compute implementation in *InternalMapWithStateDStream*. Looking at