Hey, I am not sure what is the intuitive behaviour here. As you are not applying a transformation on the feedback stream but pass it to a closeWith method, I thought it was somehow nature that it gets the partitioning of the iteration input, but maybe its not intuitive.
If others also think that preserving feedback partitioning should be the default I am not against it :) Btw, this still won't make it very simple. We still need as many source/sink pairs as we have different parallelism among the head operators. Otherwise the forwarding logic wont work. Cheers, Gyula Aljoscha Krettek <aljos...@apache.org> ezt írta (időpont: 2015. júl. 31., P, 11:52): > Hi, > I'm currently working on making the StreamGraph generation more centralized > (i.e. not spread across the different API classes). The question is now why > we need to switch to preserve partitioning? Could we not make "preserve" > partitioning the default and if users want to have shuffle partitioning or > anything they have to specify it manually when adding the feedback edge? > > This would make for a very simple scheme where the iteration sources are > always connected to the heads using "forward" and the tails are connected > to the iteration sinks using whatever partitioner was set by the user. This > would make it more transparent than the current default of the "shuffle" > betweens tails and iteration sinks. > > Cheers, > Aljoscha > > P.S. I now we had quite some discussion about introducing "preserve > partitioning" but now, when I think of it it should be the default... :D >
