[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12325: --- Resolution: Fixed Fix Version/s: 2.7.4 Status: Resolved (was: Patch Available) I verified test failures and pushed to branch-2.7. > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: Callers of WritableRpcEngine.call.png, > HADOOP-12325-branch-2.7.00.patch, HADOOP-12325.001.patch, > HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, > HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12325: --- Status: Patch Available (was: Reopened) > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: Callers of WritableRpcEngine.call.png, > HADOOP-12325-branch-2.7.00.patch, HADOOP-12325.001.patch, > HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, > HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HADOOP-12325: --- Attachment: HADOOP-12325-branch-2.7.00.patch > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: Callers of WritableRpcEngine.call.png, > HADOOP-12325-branch-2.7.00.patch, HADOOP-12325.001.patch, > HADOOP-12325.002.patch, HADOOP-12325.003.patch, HADOOP-12325.004.patch, > HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, HADOOP-12325.006.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-12325: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) Thanks [~anu] for the contribution and [~ajisakaa] for the review. I've commit the change to trunk and branch-2. > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: 2.8.0 > > Attachments: Callers of WritableRpcEngine.call.png, > HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, > HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch, > HADOOP-12325.006.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[
https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-12325:
--
Attachment: HADOOP-12325.006.patch
[~ajisakaa] Thanks for your review and changes to the test file. Please see my
comments below
bq. 1. Would you add a whitespace before "took " in the log message?
fixed.
bq. 2. After running the regression test locally, I can't see any logs about
sleep RPC.
On my machine if I open the file
org.apache.hadoop.ipc.TestProtoBufRpc-output.txt in the sure-fire reports
directory, I am able to see the following line.
{code}
2015-08-24 10:52:16,713 WARN ipc.Server (Server.java:logSlowRpcCalls(438)) -
Slow RPC : sleep took 3004 milliseconds to process from client 10.0.1.35:57223
{code}
bq. Attaching a patch to verify that the slow call is logged. Now the test
fails.
With the new call {code} long after = getLongCounter("RpcSlowCalls",
rpcMetrics); {code} somehow the mocking layer is still returning the old
snap-shotted value. I have modified the tests to call server layer directly
and tests are now behaving as expected.
> RPC Metrics : Add the ability track and log slow RPCs
> -
>
> Key: HADOOP-12325
> URL: https://issues.apache.org/jira/browse/HADOOP-12325
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, metrics
>Affects Versions: 2.7.1
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: Callers of WritableRpcEngine.call.png,
> HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch,
> HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch,
> HADOOP-12325.006.patch
>
>
> This JIRA proposes to add a counter called RpcSlowCalls and also a
> configuration setting that allows users to log really slow RPCs. Slow RPCs
> are RPCs that fall at 99th percentile. This is useful to troubleshoot why
> certain services like name node freezes under heavy load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-12325: --- Attachment: HADOOP-12325.005.test.patch Attaching a patch to verify that the slow call is logged. Now the test fails. > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: Callers of WritableRpcEngine.call.png, > HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, > HADOOP-12325.004.patch, HADOOP-12325.005.patch, HADOOP-12325.005.test.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-12325: -- Status: Patch Available (was: Open) > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: Callers of WritableRpcEngine.call.png, > HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, > HADOOP-12325.004.patch, HADOOP-12325.005.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HADOOP-12325: -- Status: Open (was: Patch Available) > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: Callers of WritableRpcEngine.call.png, > HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, > HADOOP-12325.004.patch, HADOOP-12325.005.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-12325: -- Attachment: HADOOP-12325.005.patch Support Slow RPC logging for WriteableRpcEngine also > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: Callers of WritableRpcEngine.call.png, > HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, > HADOOP-12325.004.patch, HADOOP-12325.005.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-12325: -- Attachment: HADOOP-12325.004.patch fix java doc > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: Callers of WritableRpcEngine.call.png, > HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch, > HADOOP-12325.004.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HADOOP-12325: Attachment: Callers of WritableRpcEngine.call.png > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: Callers of WritableRpcEngine.call.png, > HADOOP-12325.001.patch, HADOOP-12325.002.patch, HADOOP-12325.003.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[
https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HADOOP-12325:
--
Attachment: HADOOP-12325.003.patch
Fixed CheckStyle issues , two issues still remain, please ignore them.
* {{Server.java}} - File is too long
* Variable 'logSlowRPC' must be private - it is a hadoop metric and follows
the general pattern in the file.
> RPC Metrics : Add the ability track and log slow RPCs
> -
>
> Key: HADOOP-12325
> URL: https://issues.apache.org/jira/browse/HADOOP-12325
> Project: Hadoop Common
> Issue Type: Improvement
> Components: ipc, metrics
>Affects Versions: 2.7.1
>Reporter: Anu Engineer
>Assignee: Anu Engineer
> Attachments: HADOOP-12325.001.patch, HADOOP-12325.002.patch,
> HADOOP-12325.003.patch
>
>
> This JIRA proposes to add a counter called RpcSlowCalls and also a
> configuration setting that allows users to log really slow RPCs. Slow RPCs
> are RPCs that fall at 99th percentile. This is useful to troubleshoot why
> certain services like name node freezes under heavy load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-12325: -- Attachment: HADOOP-12325.002.patch Thanks for detailed review [~xyao] . I have attached a revised patch. Please see below for my detailed comments. bq. 1. Do you miss updating all the caller of ProtobufRpcEngine.call() to pass receiveTime using Time.monotonicNow() instead of the Time.now()? fixed, I have reverted to using Time.now() for in this patch. bq. 2. Do we need update WritableRpcEngine.java class with logSlowRpcCalls()? I could not find any place where we were using WritableRpcEngine for real, hence I did not make that change. bq. 3. NIT: Can you put the magic number 1024 as final variable like fixed bq. 4. Can you change the following from fixed bq. 5. NIT: Rpc -> RPC to be consistent fixed bq. Can you make the SleepRequestProto accepting a duration parameter instead of the fixed SLEEP_DURATION (1000ms)? done bq. 7. Is it possible to test with 1K fast calls instead of 10K calls to save test resources without affecting the results? I had benchmarked these calls and even with 10K it is in milliseconds. The reason I was making 10 K calls is to make sure that the test exercises the computation and the statistical significance properly. > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HADOOP-12325.001.patch, HADOOP-12325.002.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-12325: -- Status: Patch Available (was: Open) > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HADOOP-12325.001.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-12325) RPC Metrics : Add the ability track and log slow RPCs
[ https://issues.apache.org/jira/browse/HADOOP-12325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HADOOP-12325: -- Attachment: HADOOP-12325.001.patch This patch adds : * An metric called RpcSlowCalls * Ability to log Slow calls if ipc.server.log.slow.rpc is set to true > RPC Metrics : Add the ability track and log slow RPCs > - > > Key: HADOOP-12325 > URL: https://issues.apache.org/jira/browse/HADOOP-12325 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc, metrics >Affects Versions: 2.7.1 >Reporter: Anu Engineer >Assignee: Anu Engineer > Attachments: HADOOP-12325.001.patch > > > This JIRA proposes to add a counter called RpcSlowCalls and also a > configuration setting that allows users to log really slow RPCs. Slow RPCs > are RPCs that fall at 99th percentile. This is useful to troubleshoot why > certain services like name node freezes under heavy load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
