[
https://issues.apache.org/jira/browse/FLINK-25029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450509#comment-17450509
]
Arvid Heise commented on FLINK-25029:
-------------------------------------
Just FYI, the HDFS Filesystem lives in a separate class loader. It has access
to all Flink core classes but it would be difficult to add a context class just
to the Filesystem.
But I haven't understood which components are involved. Does it also apply to
hive-specific things?
Note also that the filesystem will not have access to any job-specific
information; that's why I thought it has to be added to the coordinator. I
could imagine that we need to add a general context facility (e.g. service
interface on JobMaster and service implementation in filesystem).
Be also aware that we constantly need to swap context classloaders for various
things. So this is not a reliable hook. Thread locals seem to be a good way at
first glance.
> Hadoop Caller Context Setting In Flink
> --------------------------------------
>
> Key: FLINK-25029
> URL: https://issues.apache.org/jira/browse/FLINK-25029
> Project: Flink
> Issue Type: Improvement
> Components: FileSystems
> Reporter: 刘方奇
> Assignee: 刘方奇
> Priority: Major
>
> For a given HDFS operation (e.g. delete file), it's very helpful to track
> which upper level job issues it. The upper level callers may be specific
> Oozie tasks, MR jobs, and hive queries. One scenario is that the namenode
> (NN) is abused/spammed, the operator may want to know immediately which MR
> job should be blamed so that she can kill it. To this end, the caller context
> contains at least the application-dependent "tracking id".
> The above is the main effect of the Caller Context. HDFS Client set Caller
> Context, then name node get it in audit log to do some work.
> Now the Spark and hive have the Caller Context to meet the HDFS Job Audit
> requirement.
> In my company, flink jobs often cause some problems for HDFS, so we did it
> for preventing some cases.
> If the feature is general enough. Should we support it, then I can submit a
> PR for this.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
