[ https://issues.apache.org/jira/browse/FLINK-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159232#comment-15159232 ]
ASF GitHub Bot commented on FLINK-3315: --------------------------------------- Github user tillrohrmann commented on a diff in the pull request: https://github.com/apache/flink/pull/1641#discussion_r53816021 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SlotAllocationTest.java --- @@ -40,17 +48,142 @@ public void test() { public boolean filter(Long value) { return false; } }; - env.generateSequence(1, 10).filter(dummyFilter).isolateResources().filter(dummyFilter) - .disableChaining().filter(dummyFilter).startNewResourceGroup().filter(dummyFilter) - .startNewChain().print(); + env.generateSequence(1, 10) + .filter(dummyFilter).slotSharingGroup("isolated") + .filter(dummyFilter).slotSharingGroup("default").disableChaining() + .filter(dummyFilter).slotSharingGroup("group 1") + .filter(dummyFilter).startNewChain() + .print().disableChaining(); + + // verify that a second pipeline does not inherit the groups from the first pipeline + env.generateSequence(1, 10) + .filter(dummyFilter).slotSharingGroup("isolated-2") + .filter(dummyFilter).slotSharingGroup("default").disableChaining() + .filter(dummyFilter).slotSharingGroup("group 2") + .filter(dummyFilter).startNewChain() + .print().disableChaining(); JobGraph jobGraph = env.getStreamGraph().getJobGraph(); List<JobVertex> vertices = jobGraph.getVerticesSortedTopologicallyFromSources(); - assertEquals(vertices.get(0).getSlotSharingGroup(), vertices.get(2).getSlotSharingGroup()); + assertEquals(vertices.get(0).getSlotSharingGroup(), vertices.get(3).getSlotSharingGroup()); + assertNotEquals(vertices.get(0).getSlotSharingGroup(), vertices.get(2).getSlotSharingGroup()); + assertNotEquals(vertices.get(3).getSlotSharingGroup(), vertices.get(4).getSlotSharingGroup()); + assertEquals(vertices.get(4).getSlotSharingGroup(), vertices.get(5).getSlotSharingGroup()); + assertEquals(vertices.get(5).getSlotSharingGroup(), vertices.get(6).getSlotSharingGroup()); --- End diff -- These conditions are actually not so easy to verify since you have to know how the vertices are internally ordered. > Fix Slot Sharing in Streaming API > --------------------------------- > > Key: FLINK-3315 > URL: https://issues.apache.org/jira/browse/FLINK-3315 > Project: Flink > Issue Type: Improvement > Components: Streaming > Affects Versions: 1.0.0 > Reporter: Aljoscha Krettek > Assignee: Aljoscha Krettek > Priority: Blocker > > Right now, the slot sharing/resource group logic is a bit "nebulous". The > slot sharing group that operators are put in depends on the order in which > operations are created. For example, in this case: > {code} > Source a = env.source() > Source b = env.source() > a.map().startNewResourceGroup().sink() > b.map().sink() > {code} > We end up with two resource groups: > - group 1: source a > - group 2: map(), sink(), source b, map(), sink() > The reason is that the slot sharing id is incremented when transforming the > {{startNewResouceGroup()}} call and all operators that are transformed > afterwards in graph traversal get that new slot sharing id. > (There is also {{isolateResources()}} which can be used to isolate an > operator.) > What I propose is to remove {{startNewResourceGroup()}} and > {{isolateResouces()}} and replace it with {{slotSharingGroup(String)}}. By > default, operations would be in slot sharing group "default". This allows > very fine grained control over what operators end up in which slot sharing > group. For example, I could have this topology: > {code} > Source a = env.source().slotSharingGroup("sources") > Source b = env.source().slotSharingGroup("sources") > a.map().slotSharingGroup("heavy a").sink().slotSharingGroup("sinks") > b.map().slotSharingGroup("heavy b").sink().slotSharingGroup("sinks") > {code} > Which would isolate the lightweight sources and sinks in a group and put > heavy operations inside their own slot groups. > This is a bit more low level than the previous API and requires more calls > than a simple {{startNewResourceGroup()}} but I think not many people would > use this feature and this design makes it very clear what operations end up > in the same group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)