[ https://issues.apache.org/jira/browse/FLINK-26803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531595#comment-17531595 ]
Piotr Nowojski commented on FLINK-26803: ---------------------------------------- Would removing 15K out of 50K files solve the issue for you? Thats only reducing number of files by 30%. Implementation wise, same as [~akalashnikov], I'm not super happy about complicating the code. We would need some extra synchronisation between tasks and introduction of shared handles that would also conceptually complicate the system. > Merge small ChannelState file for Unaligned Checkpoint > ------------------------------------------------------ > > Key: FLINK-26803 > URL: https://issues.apache.org/jira/browse/FLINK-26803 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / Network > Reporter: fanrui > Priority: Major > > When making an unaligned checkpoint, the number of ChannelState files is > TaskNumber * subtaskNumber. For high parallelism job, it writes too many > small files. It causes high load for hdfs NN. > > In our production, a job writes more than 50K small files for each Unaligned > Checkpoint. Could we merge these files before write FileSystem? We can > configure the maximum number of files each TM can write in a single Unaligned > Checkpoint. -- This message was sent by Atlassian Jira (v8.20.7#820007)