azagrebin commented on issue #7186: [FLINK-10941] Keep slots which contain unconsumed result partitions URL: https://github.com/apache/flink/pull/7186#issuecomment-470629679 I agree with @zhijiangW. We discussed it also with @pnowojski, we might not merge it into 1.8.0 right away because we want to give it some exposure on running tests for couple of weeks, but the fix is still useful for subsequent minor releases of 1.8 and e.g. 1.7. Thinking more about delaying the releasing to moment of channel closing in happy scenario, when isEndOfPartitionEvent comes, there will probably be not so much to close apart of file handles which is not so much. Buffers are also pretty much flushed to netty and either already freed by netty or will be. Sorry for confusion, actually the current approach in PR is simpler than introduction of one more `isClosed` state. @QiLuo-BD I would suggest couple of more changes apart of using `Future<Boolean>` in task executor gateway, commented by @zhijiangW. The task executor gateway could use `NetworkEnviroment.PartitionManager` to check still registered and unreleased partitions because it is the central point to manage partitions. It would also simplify how it is queried now going through `TaskSlot` and lingering `Tasks`. Eventually, `NetworkEviroment` becomes `ShuffleService` which could decide whether producer can be released or not this way or another. We still make producer task executor depend on another consumer task executor, though, we do not see any problems with it now, I think it is still more safe to introduce an option as a feature flag. By default, task executor will wait for consumers, as this PR suggests, but users can always fallback to the previous behaviour using the option. I created a co-authored PR #7938 on top of your PR with those changes rebased on the fresh master. You could review it and we decide further.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
