liang yu created FLINK-36112: -------------------------------- Summary: Add Support for CreateFlag.NO_LOCAL_WRITE in FLINK on YARN's File Creation to Manage Disk Space and Network Load in Labeled YARN Nodes Key: FLINK-36112 URL: https://issues.apache.org/jira/browse/FLINK-36112 Project: Flink Issue Type: Improvement Reporter: liang yu
{*}Description{*}: I am currently using Apache Flink to write files into Hadoop. The Flink application runs on a labeled YARN queue. During operation, it has been observed that the local disks on these labeled nodes get filled up quickly, and the network load is significantly high. This issue arises because Hadoop prioritizes writing files to the local node first, and the number of these labeled nodes is quite limited. {*}Problem{*}: The current behavior leads to inefficient disk space utilization and high network traffic on these few labeled nodes, which could potentially affect the performance and reliability of the application. As shown in the picture, the host I circled have a average net_bytes_sent speed 1.2GB/s while the others are just 50MB/s, this imbalance in network and disk space nearly destroyed the whole cluster. {*}Implementation{*}: The implementation would involve adding a method of FileSystem.class to support the {{CreateFlag.NO_LOCAL_WRITE}} when we try to create a new file through HadoopFileSystem.create() API. What's more, I modify the code of FileSink class so that we can choose to enable no_local_write or disable this feature. This will provide flexibility to Flink running in labeled Yarn queues to opt for non-local writes when necessary. -- This message was sent by Atlassian Jira (v8.20.10#820010)