liang yu created HDFS-17577:
-------------------------------
Summary: Add Support for CreateFlag.NO_LOCAL_WRITE in File
Creation to Manage Disk Space and Network Load in Labeled YARN Nodes
Key: HDFS-17577
URL: https://issues.apache.org/jira/browse/HDFS-17577
Project: Hadoop HDFS
Issue Type: New Feature
Components: dfsclient
Reporter: liang yu
{*}Description{*}: I am currently using Apache Flink to write files into
Hadoop. The Flink application runs on a labeled YARN queue. During operation,
it has been observed that the local disks on these labeled nodes get filled up
quickly, and the network load is significantly high. This issue arises because
Hadoop prioritizes writing files to the local node first, and the number of
these labeled nodes is quite limited.
{*}Problem{*}: The current behavior leads to inefficient disk space utilization
and high network traffic on these few labeled nodes, which could potentially
affect the performance and reliability of the application.
{*}Implementation{*}: The implementation would involve adding an configuration
_dfs.client.write.no_local_write_ to support the {{CreateFlag.NO_LOCAL_WRITE}}
during the file creation process in Hadoop's file system APIs. This will
provide flexibility to applications like Flink running in labeled queues to opt
for non-local writes when necessary.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
