[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823227#comment-16823227
]
Chao Sun commented on HDFS-14406:
-
[~xuel1]: I'm still in favor of reusing the existing {{RpcDetailedMetrics}} as
most of the logic are the same between this and the proposed
{{RpcUserMetrics}}. It would be good if we can use some prefix to differentiate
the user metrics and the RPC method metrics.
> Add per user RPC Processing time
>
>
> Key: HDFS-14406
> URL: https://issues.apache.org/jira/browse/HDFS-14406
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Xue Liu
>Assignee: Xue Liu
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch,
> HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch,
> HDFS-14406.006.patch
>
>
> For a shared cluster we would want to separate users' resources, as well as
> having our metrics reflecting on the usage, latency, etc, for each user.
> This JIRA aims to add per user RPC processing time metrics and expose it via
> JMX.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822267#comment-16822267 ] Xue Liu commented on HDFS-14406: [~xkrogen] Makes sense to me, I will go back to per user metrics for this jira. Though I don't believe volume will be a blocking factor. We have metrics like NameNodeInfo via jmx give us all info on LiveNode and DeadNode, which is in the scale of thousands. And in a large system we would have parsing wherever we collect them, so I am not worried about readability either. [~csun] Since we are only talking about per user metrics here, which approach you think is better? we can pass a flag to RpcDetailedMetrics, and add different prefix accordingly, or we just create another metrics class? > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch, > HDFS-14406.006.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822258#comment-16822258
]
Chao Sun commented on HDFS-14406:
-
Thanks [~xuel1] for the update. Instead of adding a new class
{{RpcUserMetrics}}, can we just reuse the {{RpcDetailedMetrics}} by adding a
flag to it?
{code}
public void addProcessingTime(String name, int processingTime) {
String prefix = name;
if (this.userMetricsEnabled) {
UserGroupInformation ugi = Server.getRemoteUser();
String userName = ugi == null ? "null" : ugi.getUserName();
prefix = prefix + "-" + username;
}
rates.add(prefix, processingTime);
}
{code}
Then you only need to keep one instance of {{RpcDetailedMetrics}} in
{{Server}}, and code will look cleaner.
> Add per user RPC Processing time
>
>
> Key: HDFS-14406
> URL: https://issues.apache.org/jira/browse/HDFS-14406
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Xue Liu
>Assignee: Xue Liu
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch,
> HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch,
> HDFS-14406.006.patch
>
>
> For a shared cluster we would want to separate users' resources, as well as
> having our metrics reflecting on the usage, latency, etc, for each user.
> This JIRA aims to add per user RPC processing time metrics and expose it via
> JMX.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822257#comment-16822257
]
Erik Krogen commented on HDFS-14406:
I don't have time to take a full look at the patch today, but will give my 2c
on this question:
{quote}
Kept this per user per user per method metrics. What do you guys think of this
idea?
{quote}
I don't think per-user per-method metrics are a good idea. In my current
system, just wading through the overall system list of per-method metrics is a
daunting task. For installations with large user counts (100s or 1000s) this
will make these metrics completely unusable due to their huge volume.
> Add per user RPC Processing time
>
>
> Key: HDFS-14406
> URL: https://issues.apache.org/jira/browse/HDFS-14406
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Xue Liu
>Assignee: Xue Liu
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch,
> HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch,
> HDFS-14406.006.patch
>
>
> For a shared cluster we would want to separate users' resources, as well as
> having our metrics reflecting on the usage, latency, etc, for each user.
> This JIRA aims to add per user RPC processing time metrics and expose it via
> JMX.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822253#comment-16822253
]
Hadoop QA commented on HDFS-14406:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 41s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
42s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
4s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 4s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 9 new + 379 unchanged - 5 fixed = 388 total (was 384) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
43s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
37s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 47s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14406 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966519/HDFS-14406.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 663314f209ba 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 317fcba |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/26680/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/26680/artifact/out/patch-compile-root.txt
|
| javac |
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16822235#comment-16822235 ] Xue Liu commented on HDFS-14406: Hi guys thanks for the comments! Reflected on [~xkrogen]'s comment of extending, as in patch 5, I made these changes: # Make RpcUserMetrics a subclass of RpcDetailedMetrics. Only keep 2 metrics rpcMetrics and rpcDetailedMetrics, and determine whether we create RpcUserMetrics or RpcDetailedMetrics based on enable flag at runtime. # Introduce per user per method metrics, as I am receiving suggestion that this would be even more useful because it gives us an idea on how the RPC cost of each user looks. In patch 6: 1. I found out that that this dynamic type doesn't necessarily benefit our performance, but make the code look more complicated than it should be. Now I am just creating rpcUserMetrics regardless, and only update it when there is enable flag. 2. Kept this per user per user per method metrics. What do you guys think of this idea? [~thefacts] Some other fixes: 1. [~csun] I changed String.format to String plus for better performance. 2. Docs in core-default, Metrics.md are still as user metrics. I will update them once we have impl worked out. 3. A few checkstyles, I will fix them the last. > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch, > HDFS-14406.006.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16821319#comment-16821319
]
Chao Sun commented on HDFS-14406:
-
Thanks [~xuel1]. A few comments beside the above mentioned:
1. Why {{addDeferredProcessingTime}} is added in the new patch? in
{{updateMetrics}} it is called even when {{deferredCall}} is false. In
{{updateMetrics}} it is called twice.
2. {{String.format}} is used in {{addProcessingTime}}, which sits in a hot
path. Should we be concerned about the potential performance impact?
3. Like mentioned above, I think we need to implement {{RpcUserMetrics}}
better. It is sharing a fair amount of code with {{RpcDetailedMetrics}}, and
seems having a separate class for this doesn't bring much benefit. Some of the
branchings are not necessary, such as:
{code:java}
if (enableRpcUserMetrics) {
((RpcUserMetrics)rpcDetailedMetrics).shutdown();
}
else {
this.rpcDetailedMetrics.shutdown();
}
{code}
4. Can we put else in the same line as the previous }?
> Add per user RPC Processing time
>
>
> Key: HDFS-14406
> URL: https://issues.apache.org/jira/browse/HDFS-14406
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Xue Liu
>Assignee: Xue Liu
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch,
> HDFS-14406.003.patch, HDFS-14406.004.patch, HDFS-14406.005.patch
>
>
> For a shared cluster we would want to separate users' resources, as well as
> having our metrics reflecting on the usage, latency, etc, for each user.
> This JIRA aims to add per user RPC processing time metrics and expose it via
> JMX.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820632#comment-16820632
]
Hadoop QA commented on HDFS-14406:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
30s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
47s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 47s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 15 new + 357 unchanged - 5 fixed = 372 total (was 362) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
31s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
28s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14406 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12966299/HDFS-14406.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux af2708adc721 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 685cb83 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/2/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/2/artifact/out/patch-compile-root.txt
|
| javac |
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820537#comment-16820537
]
Erik Krogen commented on HDFS-14406:
Thanks for the ping, [~elgoiri]. I do have some comments:
* Creating a new {{rpcuser}} metrics context seems potentially overkill. This
is still providing details about RPC calls, so it seems to me that it can still
live under the {{rpcdetailed}} context with the sub-name "rpcuser".
* {{RpcUserMetrics}} has an {{init()}} method that initializes it with the
methods of a {{protocol}}. This is completely meaningless in this context,
since {{RpcUserMetrics}} does not store metrics about method names. This should
be removed.
* {{RpcUserMetrics}} and {{RpcDetailedMetrics}} are nearly identical, can we
have a common base class which they both inherit from, or even just let
{{RpcUserMetrics extends RpcDetailedMetrics}}? The only differences are the
init step, and the name which is passed to the registry.
* In the Javadoc for {{IPC_SERVER_USER_METRICS_DEFAULT}}, can we use a
{{@link}} tag to refer to the key?
* In the description within {{core-site.xml}}, the sentence has a typo:
{quote}
Set the whether we enable per user rpc processing time metrics on namenode rpc
server.
{quote}
should be
{quote}
Set whether we enable per user rpc processing time metrics on NameNode rpc
server.
{quote}
(An extra "the", and also IMO NameNode should be capitalized)
* Within {{Metrics.md}}, I think the descriptions could use updating:
{quote}
| *username*`NumOps` | Total number of the times RPC with username is called |
| *username*`AvgTime` | Average turn around time of the user RPC in
milliseconds |
{quote}
What does "turn around time" even mean? I would prefer:
{quote}
| *username*`NumOps` | Total number of times RPCs were called by username |
| *username*`AvgTime` | Average processing time of the user's RPCs in
milliseconds |
{quote}
> Add per user RPC Processing time
>
>
> Key: HDFS-14406
> URL: https://issues.apache.org/jira/browse/HDFS-14406
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Xue Liu
>Assignee: Xue Liu
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch,
> HDFS-14406.003.patch, HDFS-14406.004.patch
>
>
> For a shared cluster we would want to separate users' resources, as well as
> having our metrics reflecting on the usage, latency, etc, for each user.
> This JIRA aims to add per user RPC processing time metrics and expose it via
> JMX.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820522#comment-16820522 ] Íñigo Goiri commented on HDFS-14406: [^HDFS-14406.004.patch] LGTM. +1 I'll give a couple days to others ([~linyiqun], [~xkrogen], [~daryn]) to add more comments if so. > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818463#comment-16818463 ] Xue Liu commented on HDFS-14406: [~linyiqun] Thanks for the comment! Checkstyles and others has been fixed. Regarding unit test, I think all metrics registered using MetricsRegistry are exposed via JMX, do I still need to add a single JMX test for it? > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch, > HDFS-14406.003.patch, HDFS-14406.004.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818446#comment-16818446
]
Hadoop QA commented on HDFS-14406:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 22m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
28s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14406 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965994/HDFS-14406.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 973d96d07acf 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5583e1b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26637/testReport/ |
| Max. process+thread count | 1347 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26637/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add per user
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818355#comment-16818355
]
Hadoop QA commented on HDFS-14406:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
47s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 47s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
31s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 0m
27s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14406 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965985/HDFS-14406.004.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 99a8a1b70cd5 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 254efc9 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/26635/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/26635/artifact/out/patch-compile-root.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/26635/artifact/out/patch-compile-root.txt
|
| mvnsite |
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815918#comment-16815918
]
Yiqun Lin commented on HDFS-14406:
--
This idea makes sense to me, although here the process time contains the
lock-waiting time not pure rpc process time. But in some scenarios, people also
want to know this time in metrics. We make this optional now, so it looks fine.
Some review comments:
* RpcUserMetrics*
1.We should unregister the MetricsRegistry instance when shutdown
{code}
/**
* Shutdown the instrumentation for the process.
*/
public void shutdown() {
DefaultMetricsSystem.instance().unregisterSource(name);
}
{code}
2. Can we complete the java doc for the function {{addDeferredProcessingTime}}?
*CommonConfigurationKeysPublic*
Document the new config {{ipc.server.user.metrics}} in core-default.xml.
I don't see the test for getting user RPC metric via JMX. Can you add related
test for this?
> Add per user RPC Processing time
>
>
> Key: HDFS-14406
> URL: https://issues.apache.org/jira/browse/HDFS-14406
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Xue Liu
>Assignee: Xue Liu
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch,
> HDFS-14406.003.patch
>
>
> For a shared cluster we would want to separate users' resources, as well as
> having our metrics reflecting on the usage, latency, etc, for each user.
> This JIRA aims to add per user RPC processing time metrics and expose it via
> JMX.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815868#comment-16815868
]
Hadoop QA commented on HDFS-14406:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 13s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 7 new + 374 unchanged - 0 fixed = 381 total (was 374) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 26s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}111m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14406 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965652/HDFS-14406.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2a3100033657 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / fcc7f9b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26617/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26617/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26617/testReport/ |
| Max. process+thread count | 1347 (vs. ulimit of 1) |
|
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815829#comment-16815829
]
Xue Liu commented on HDFS-14406:
Fixed some checkstyles. Some is left unfixed, such as:
{quote}./hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcUserMetrics.java:41:
final MetricsRegistry registry;:25: Variable 'registry' must be private and
have accessor methods. [VisibilityModifier]
{quote}
to maintain consistency with other rpc metrics.
> Add per user RPC Processing time
>
>
> Key: HDFS-14406
> URL: https://issues.apache.org/jira/browse/HDFS-14406
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: Xue Liu
>Assignee: Xue Liu
>Priority: Minor
> Fix For: 3.2.0
>
> Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch,
> HDFS-14406.003.patch
>
>
> For a shared cluster we would want to separate users' resources, as well as
> having our metrics reflecting on the usage, latency, etc, for each user.
> This JIRA aims to add per user RPC processing time metrics and expose it via
> JMX.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814977#comment-16814977
]
Hadoop QA commented on HDFS-14406:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
54s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 9 new + 374 unchanged - 0 fixed = 383 total (was 374) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 42s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ha.TestZKFailoverController |
| | hadoop.util.TestReadWriteDiskValidator |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14406 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965530/HDFS-14406.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5dc274ffae59 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8740755 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26610/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/26610/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26610/testReport/ |
| Max.
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16814938#comment-16814938 ] Xue Liu commented on HDFS-14406: Thanks for the comments and suggestions! New patch available, makes the user RPC metrics optional. Agree on that we are now facing the bottleneck of lock waiting time, that would have an unfair effect on per user RPC processing time. In this case, the difference of RPC processing time may not be that big. However I do like to think of this metric as general indicator, that we can probably expose to users. Also, it might be good we have some generosity here, so that other ipc like Yarn may also use it. [~elgoiri][~daryn] Thanks for the suggestion, I have fixed checkstyles and made it optional. [~xkrogen], the work on cost-based FCQ seems interesting, which is something we also want in our prod cluster in the future. From your perspective, would it be interesting if we can have a more detailed metrics, say per user per method rpc processing time? > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch, HDFS-14406.002.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812511#comment-16812511 ] Erik Krogen commented on HDFS-14406: I'd like to echo [~daryn]'s sentiment that this probably won't provide much useful information for HDFS given the lock behavior. You can see more discussion of this in HDFS-14403, a recently updated proposal to extend the FairCallQueue to consider cost of an operation (in the case of HDFS, this is considered to be lock hold time). The design document has some extended discussion of why looking at the overall processing time is insufficient for HDFS. In that patch, we've surfaced information about the lock hold time itself up to the RPC layer; maybe it would make more sense to surface that value rather than the processing time? [~daryn], I'd love to learn more about your patch for excluding lock wait time from processing time. > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812451#comment-16812451 ] Daryn Sharp commented on HDFS-14406: I'd also prefer this be optional. The overhead will provide little of the intended value if intended for hdfs. Processing time for hdfs ops includes lock wait time which unfairly inflates processing time for all ops. Ie. If one user performs many write operations, all users are billed for the lock wait time while the write ops complete. (Note: this may change soon if my internal patch to exclude lock wait from processing time for the decay scheduler proves successful) Other than that, the following: "One metrics is exposed" should be "A metric is exposed". > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811457#comment-16811457 ] Íñigo Goiri commented on HDFS-14406: A +1 from Yetus, impressive. Let's fix the checkstyles that don't break consistency. > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[
https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811434#comment-16811434
]
Hadoop QA commented on HDFS-14406:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 12 new + 261 unchanged - 0 fixed = 273 total (was 261) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
39s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14406 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12965060/HDFS-14406.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 866d2c5debe6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1b9ba0e |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26591/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26591/testReport/ |
| Max. process+thread count | 1605 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26591/console |
| Powered by | Apache Yetus 0.8.0
[jira] [Commented] (HDFS-14406) Add per user RPC Processing time
[ https://issues.apache.org/jira/browse/HDFS-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16811341#comment-16811341 ] Xue Liu commented on HDFS-14406: Patch available. Just want to say being a new contributor and all, thank you advance for reviewing and commenting :) > Add per user RPC Processing time > > > Key: HDFS-14406 > URL: https://issues.apache.org/jira/browse/HDFS-14406 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Xue Liu >Assignee: Xue Liu >Priority: Minor > Fix For: 3.2.0 > > Attachments: HDFS-14406.001.patch > > > For a shared cluster we would want to separate users' resources, as well as > having our metrics reflecting on the usage, latency, etc, for each user. > This JIRA aims to add per user RPC processing time metrics and expose it via > JMX. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
