Matthias Pohl created FLINK-32098: ------------------------------------- Summary: Dispatcher#submitJob calls Dispatcher#isInGloballyTerminalState up to three times which might be expensive due to IO Key: FLINK-32098 URL: https://issues.apache.org/jira/browse/FLINK-32098 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.16.1, 1.17.0, 1.18.0 Reporter: Matthias Pohl
{{Dispatcher#submitJob}} calls {{Dispatcher#isInGloballyTerminalState}} up to three times (1x through {{Dispatcher#isDuplicateJob}} and 2x directly) which calls {{JobResultStore#hasJobResultStore}}. {{hasJobResultStore}} calls {{hasDirtyJobResultEntry}} and {{hasCleanJobResultEntry}} if the underlying job hasn't completed globally, yet. Both calls run {{FileSystem#exists}} on an non-existing file which can be a quite expensive operation (depending on the {{FileSystem}} implementation for object storage) since it might require a full table scan. tbh, so far, nobody complained. But we might want to either reconsider the {{FileSystemJobResultStore}}/{{JobResultStore#hasJobResultEntry}} implementation or, at least, reduce the number of {{isInGloballyTerminalState}} in the {{Dispatcher}} and document the performance issue in the JavaDoc. -- This message was sent by Atlassian Jira (v8.20.10#820010)