[jira] [Commented] (HDFS-13293) RBF: The RouterRPCServer should transfer client ip via CallerContext to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209490#comment-17209490 ] Hui Fei commented on HDFS-13293: Modify the caption and the description, focus on audit log > RBF: The RouterRPCServer should transfer client ip via CallerContext to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13293.001.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > Otherwise, the namenode don't know the client's callerContext > This jira focuses on audit log which logs real client ip. Leave locality to > HDFS-13248 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13293) RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17210722#comment-17210722 ] Hui Fei commented on HDFS-13293: Merged. [~maobaolong] Thanks for report, [~aajisaka] [~elgoiri] Thanks for review! > RBF: The RouterRPCServer should transfer client IP via CallerContext to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Attachments: HDFS-13293.001.patch > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Otherwise, the namenode don't know the client's callerContext > This jira focuses on audit log which logs real client ip. Leave locality to > HDFS-13248 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13293) RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212912#comment-17212912 ] Chengwei Wang commented on HDFS-13293: -- Hi [~ferhui], did you test this patch while a mount table with multi-destinations? In my test cases, there were two issues when a mount table contains multi-destinations. 1. The clientIp duplicated in CallerContext when RouterRpcClient#invokeSequential. It caused by that current thread would append the clientIp to current CallerContext when it called RouterRpcClient#invokeMethod every time. {code:java} 2020-10-13 14:46:17,083 [IPC Server handler 6 on default port 13189] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(8680)) - allowed=true ugi=root (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/test_dir_with_callercontextdst=nullperm=null proto=rpc callerContext=clientContext,clientIp:10.145.43.214 2020-10-13 14:46:17,084 [IPC Server handler 1 on default port 14269] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(8680)) - allowed=true ugi=root (auth:SIMPLE) ip=/127.0.0.1 cmd=getfileinfo src=/test_dir_with_callercontextdst=nullperm=null proto=rpc callerContext=clientContext,clientIp:10.145.43.214,clientIp:10.145.43.214{code} 2. the clientIp missed in CallerContext when RouterRpcClient#invokeConcurrent. It caused by that RouterRpcClient#invokeConcurrent used a ThreadPoolExecutor to execute request concurrent, however the worker threads of ThreadPoolExecutor can't get clientIp by Server#getRemoteAddress (thread local value only initialized of current thread). {code:java} 2020-10-13 14:46:18,305 [IPC Server handler 7 on default port 13189] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(8680)) - allowed=true ugi=root (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/ dst=null perm=null proto=rpc callerContext=clientContext 2020-10-13 14:46:18,305 [IPC Server handler 0 on default port 14269] INFO FSNamesystem.audit (FSNamesystem.java:logAuditMessage(8680)) - allowed=true ugi=root (auth:SIMPLE) ip=/127.0.0.1 cmd=listStatus src=/ dst=null perm=null proto=rpc callerContext=clientContext {code} Please help to check the issues, and I would try to fix these if necessary. > RBF: The RouterRPCServer should transfer client IP via CallerContext to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HDFS-13293.001.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Otherwise, the namenode don't know the client's callerContext > This jira focuses on audit log which logs real client ip. Leave locality to > HDFS-13248 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13293) RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212944#comment-17212944 ] Hui Fei commented on HDFS-13293: [~smarthan] Thanks for report! Would you please file a new jira and Upload your improve fix? Thanks > RBF: The RouterRPCServer should transfer client IP via CallerContext to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: maobaolong >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HDFS-13293.001.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Otherwise, the namenode don't know the client's callerContext > This jira focuses on audit log which logs real client ip. Leave locality to > HDFS-13248 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13293) RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17213026#comment-17213026 ] Chengwei Wang commented on HDFS-13293: -- Hi [~ferhui], I have filed a new jira HDFS-15630 and would try to fix the issue soon. > RBF: The RouterRPCServer should transfer client IP via CallerContext to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: maobaolong >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: HDFS-13293.001.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Otherwise, the namenode don't know the client's callerContext > This jira focuses on audit log which logs real client ip. Leave locality to > HDFS-13248 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13293) RBF: The RouterRPCServer should transfer client IP via CallerContext to NamenodeRpcServer
[ https://issues.apache.org/jira/browse/HDFS-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17692551#comment-17692551 ] Takanobu Asanuma commented on HDFS-13293: - I added 3.3.5 to fix versions. > RBF: The RouterRPCServer should transfer client IP via CallerContext to > NamenodeRpcServer > - > > Key: HDFS-13293 > URL: https://issues.apache.org/jira/browse/HDFS-13293 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Baolong Mao >Assignee: Hui Fei >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > Attachments: HDFS-13293.001.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > Otherwise, the namenode don't know the client's callerContext > This jira focuses on audit log which logs real client ip. Leave locality to > HDFS-13248 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org