[
https://issues.apache.org/jira/browse/FLINK-25029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448489#comment-17448489
]
刘方奇 commented on FLINK-25029:
-----------------------------
[~dmvk] ,additionally, "call `CallerContext#setCurrent` before initiating any
calls that interact with HDFS" seems very difficult.
Actually, the detail of the implement is that the thread store the caller
context with a InheritableThreadLocal.So all the child threads can share this
state.Whenever child threads use the hadoop filesystem api, hadoop client will
check whether the InheritableThreadLocal hold the caller context, then send it
to name node if the value is not null.
> Hadoop Caller Context Setting In Flink
> --------------------------------------
>
> Key: FLINK-25029
> URL: https://issues.apache.org/jira/browse/FLINK-25029
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
> Reporter: 刘方奇
> Priority: Major
>
> For a given HDFS operation (e.g. delete file), it's very helpful to track
> which upper level job issues it. The upper level callers may be specific
> Oozie tasks, MR jobs, and hive queries. One scenario is that the namenode
> (NN) is abused/spammed, the operator may want to know immediately which MR
> job should be blamed so that she can kill it. To this end, the caller context
> contains at least the application-dependent "tracking id".
> The above is the main effect of the Caller Context. HDFS Client set Caller
> Context, then name node get it in audit log to do some work.
> Now the Spark and hive have the Caller Context to meet the HDFS Job Audit
> requirement.
> In my company, flink jobs often cause some problems for HDFS, so we did it
> for preventing some cases.
> If the feature is general enough. Should we support it, then I can submit a
> PR for this.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
