Fwd: FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？
Yes，that’s the point , FilesCreated = CreateFileOps + FsDirMkdirOp All I can say is --thanksDarling Andrew D.Lin下面是被转发的邮件：发件人: Yun Tang <myas...@live.com>主题: 回复： FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？日期: 2019年7月23日 GMT+8 下午4:05:42收件人: 陈Darling <chendonglin...@gmail.com>, "qcx978132...@gmail.com" <qcx978132...@gmail.com>抄送: "user@flink.apache.org" <user@flink.apache.org>Hi AndrewFilesCreated = CreateFileOps + FsDirMkdirOp Please refer to [1] and [2] to know the meaning of this metrics.[1] https://github.com/apache/hadoop/blob/377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirMkdirOp.java#L178[2] https://github.com/apache/hadoop/blob/377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java#L799BestYun TangFrom: 陈Darling <chendonglin...@gmail.com>Sent: Tuesday, July 23, 2019 11:32To: qcx978132...@gmail.com <qcx978132...@gmail.com>Cc: user@flink.apache.org <user@flink.apache.org>; myas...@live.com <myas...@live.com>Subject: Fwd: FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？ Hi Yun TangYour suggestion is very very important to us. According to your suggestion, We have suggested that users increase the interval time (1 to 5 minutes) and set state.backend.fs.memory-threshold=10k. But we only have one hdfs cluster, we try to reduce Hdfs api call, I don't know if there is any possibility of re-optimization，Thank you very much for your patience and help. Darling Andrew D.Lin下面是被转发的邮件：发件人: Congxian Qiu <qcx978132...@gmail.com>主题: 回复： FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？日期: 2019年7月23日 GMT+8 上午9:48:05收件人: 陈Darling <chendonglin...@gmail.com>抄送: user <user@flink.apache.org>Hi AndrewThese API calls are for checkpoint file created/deleted, and there is an ongoing issue[1] which want to reduce the number.[1] https://issues.apache.org/jira/browse/FLINK-11696Best,Congxian陈Darling <chendonglin...@gmail.com> 于2019年7月22日周一 下午11:22写道：HiWe use ‘FsStateBackend' as our state beckend !The following figure shows the frequency of the hdfs API call.I don’t understand FilesCreated and FileDeleted is for what? All of these are necessary? Is it possible to reduce some unnecessary？Darling Andrew D.Lin
Re: FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？
Hi Andrew FilesCreated = CreateFileOps + FsDirMkdirOp Please refer to [1] and [2] to know the meaning of this metrics. [1] https://github.com/apache/hadoop/blob/377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirMkdirOp.java#L178 [2] https://github.com/apache/hadoop/blob/377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java#L799 Best Yun Tang From: 陈Darling Sent: Tuesday, July 23, 2019 11:32 To: qcx978132...@gmail.com Cc: user@flink.apache.org ; myas...@live.com Subject: Fwd: FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？ Hi Yun Tang Your suggestion is very very important to us. According to your suggestion, We have suggested that users increase the interval time (1 to 5 minutes) and set state.backend.fs.memory-threshold=10k. But we only have one hdfs cluster, we try to reduce Hdfs api call, I don't know if there is any possibility of re-optimization， Thank you very much for your patience and help. Darling Andrew D.Lin 下面是被转发的邮件： 发件人: Congxian Qiu mailto:qcx978132...@gmail.com>> 主题: 回复： FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？ 日期: 2019年7月23日 GMT+8 上午9:48:05 收件人: 陈Darling mailto:chendonglin...@gmail.com>> 抄送: user mailto:user@flink.apache.org>> Hi Andrew These API calls are for checkpoint file created/deleted, and there is an ongoing issue[1] which want to reduce the number. [1] https://issues.apache.org/jira/browse/FLINK-11696 Best, Congxian 陈Darling mailto:chendonglin...@gmail.com>> 于2019年7月22日周一 下午11:22写道： Hi We use ‘FsStateBackend' as our state beckend ! The following figure shows the frequency of the hdfs API call. I don’t understand FilesCreated and FileDeleted is for what? All of these are necessary? Is it possible to reduce some unnecessary？ [cid:9b42fd64-e726-47f2-a745-092f6a24d62e@namprd14.prod.outlook.com] Darling Andrew D.Lin
Fwd: FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？
Hi Yun Tang Your suggestion is very very important to us. According to your suggestion, We have suggested that users increase the interval time (1 to 5 minutes) and set state.backend.fs.memory-threshold=10k. But we only have one hdfs cluster, we try to reduce Hdfs api call, I don't know if there is any possibility of re-optimization，Thank you very much for your patience and help. Darling Andrew D.Lin 下面是被转发的邮件：发件人: Congxian Qiu <qcx978132...@gmail.com>主题: 回复： FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？日期: 2019年7月23日 GMT+8 上午9:48:05收件人: 陈Darling <chendonglin...@gmail.com>抄送: user <user@flink.apache.org>Hi AndrewThese API calls are for checkpoint file created/deleted, and there is an ongoing issue[1] which want to reduce the number.[1] https://issues.apache.org/jira/browse/FLINK-11696Best,Congxian陈Darling <chendonglin...@gmail.com> 于2019年7月22日周一 下午11:22写道：HiWe use ‘FsStateBackend' as our state beckend !The following figure shows the frequency of the hdfs API call.I don’t understand FilesCreated and FileDeleted is for what? All of these are necessary? Is it possible to reduce some unnecessary？Darling Andrew D.Lin
Fwd: FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？
Hi Yun Tang Your suggestion is very very important to us. According to your suggestion, We have suggested that users increase the interval time (1 to 5 minutes) and set state.backend.fs.memory-threshold=10k. But we only have one hdfs cluster, we try to reduce Hdfs api call, I don't know if there is any possibility of re-optimization，Thank you very much for your patience and help. Darling Andrew D.Lin 下面是被转发的邮件：发件人: Congxian Qiu <qcx978132...@gmail.com>主题: 回复： FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？日期: 2019年7月23日 GMT+8 上午9:48:05收件人: 陈Darling <chendonglin...@gmail.com>抄送: user <user@flink.apache.org>Hi AndrewThese API calls are for checkpoint file created/deleted, and there is an ongoing issue[1] which want to reduce the number.[1] https://issues.apache.org/jira/browse/FLINK-11696Best,Congxian陈Darling <chendonglin...@gmail.com> 于2019年7月22日周一 下午11:22写道：HiWe use ‘FsStateBackend' as our state beckend !The following figure shows the frequency of the hdfs API call.I don’t understand FilesCreated and FileDeleted is for what? All of these are necessary? Is it possible to reduce some unnecessary？Darling Andrew D.Lin
Fwd: FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？
Hi In my understanding，CreateFile and FileCreated api is different，FileCreated is more like a check api, but I don’t find where it was called in the src source. I don’t understand when FileCreated Api was called and for what。Is FileCreated api a hdfs internal confirmation api?FLINK-11696 is to reduce CreateFile api by reducing mkdir. Will FileCreated Api be reduced?Is there anything wrong with me here?Darling Andrew D.Lin下面是被转发的邮件：发件人: Congxian Qiu <qcx978132...@gmail.com>主题: 回复： FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？日期: 2019年7月23日 GMT+8 上午9:48:05收件人: 陈Darling <chendonglin...@gmail.com>抄送: user <user@flink.apache.org>Hi AndrewThese API calls are for checkpoint file created/deleted, and there is an ongoing issue[1] which want to reduce the number.[1] https://issues.apache.org/jira/browse/FLINK-11696Best,Congxian陈Darling <chendonglin...@gmail.com> 于2019年7月22日周一 下午11:22写道：HiWe use ‘FsStateBackend' as our state beckend !The following figure shows the frequency of the hdfs API call.I don’t understand FilesCreated and FileDeleted is for what? All of these are necessary? Is it possible to reduce some unnecessary？Darling Andrew D.Lin
