[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15537392#comment-15537392
]
Erik Krogen commented on HDFS-10872:
Attaching v000 patch.
This makes {{FSNamesystemLock}} into a {{MetricsSource}}. When the lock is
released, the amount of time for which the lock was held is recorded as a
metric corresponding to the specific operation which caused the lock to be
held. This was achieved by manually specifying the operation in calls to
{{FSNamesystemLock.read/writeLock}} since there is no easy way to tell what
operation caused the lock's acquisition from within {{FSNamesystemLock}}. Lock
releases which do not specify an operation are filed under "OTHER". There are a
number of lock usages within {{BlockManager}} which have this classification
because {{BlockManager}} only views the {{Namesystem}} API, not
{{FSNamesystem}}, and I did not want to modify that API. I am open to
suggestions here.
Since this occurs along the critical path and may add overhead, it is disabled
by default. To minimize overhead, each thread stores a local copy of the
metrics. Every so often, set by a configurable interval, each thread will
insert the metrics it has accumulated into a shared {{MetricsRegistry}}. I am
doing some microbenchmark performance testing now and will post results when
they are available.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572775#comment-15572775
]
Zhe Zhang commented on HDFS-10872:
--
Thanks Erik for the patch! The structure looks good overall. A few comments:
# Since block reporting is a major factor of write locking, I think we should
add the new unlock API to {{BlockManager}}. We are not removing any existing
APIs so compatibility shouldn't be a concern. Due the size of the current patch
we can probably do that in a separate JIRA though.
# Interesting that we are almost aligned with how audit logs are recorded. On
the one hand this confirms that there's probably no easier way to get the
method name. On the other hand we should construct a String variable that can
be reused by audit logging and this locking metrics logic. This avoids one
temporary variable for each method call.
# Have you considered integrating the Metrics logic into {{NameNodeMetrics}}?
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583197#comment-15583197
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 6 new + 830 unchanged - 0 fixed = 836 total (was 830) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
49s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 2s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Should
org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock$LocalMetric be a
_static_ inner class? At FSNamesystemLock.java:inner class? At
FSNamesystemLock.java:[lines 359-376] |
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833771/HDFS-10872.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux ae23194037c7 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ed9fcbe |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17183/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17183/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
|
| unit |
https://builds.ap
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15583802#comment-15583802
]
Zhe Zhang commented on HDFS-10872:
--
Thanks Erik for the updated patch. I think we are pretty close.
About the main change in {{FSNamesystemLock}}
# If a thread lives shorter than the configured {{metricAggregationInterval}},
are we gonna lose its locking metrics? In reality it's probably not a big
concern since RPC handler threads are reused. But putting a comment here for
more thoughts.
# Selecting a _real_ default aggregation interval is not easy. Maybe we should
document it in {{hdfs-default.xml}}. Alternatively we can have 2 config knobs,
one binary and one integer.
# Agreed that the overhead with the current implementation is pretty small (say
with 10 sec interval). As a follow-on optimizaiton (depending on experience
from production deployment) maybe we can consider a combination of
hard-deadline and opportunistic try-and-backoff. E.g. at a higher frequency
than {{metricAggregationInterval}} we can try the lock for
{{opHoldtimeMetrics}}; if lock is free, dump the metrics, otherwise try lock in
some more time.
A few minors about naming:
# The below mapping merges the {{yield()}} locking time into the
{{contentSummary}} category. It looks a reasonable approximation to me. But
more opinions would be helpful.
{code}
@@ -115,7 +115,7 @@ public boolean yield() {
// unlock
dir.readUnlock();
-fsn.readUnlock();
+fsn.readUnlock("contentSummary");
{code}
# {{getBlockLocations}} currently appears as {{open}} in audit logging. It
could be an existing bug. But since it has been audit logged that way it
doesn't make sense to change here. I'll create a separate JIRA to discuss.
# {{writeUnlock("clearCorruptLazyPersistFile");}} should be
"clearCorruptLazyPersistFiles"
# Maybe the {{checkLease}} category should be {{leasesMonitor}}? There's a
one-off {{checkLease()}} method in FSN
# The one in {{NamenodeFsck#getBlockLocations}}, maybe we should use
"fsckGetBlockLocations" to differentiate from regular {{getBlockLocations}} RPC
call?
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586193#comment-15586193
]
Erik Krogen commented on HDFS-10872:
[~zhz] thanks for the detailed review!
1. Yes, if it lives shorter than the aggregation interval, its metrics will be
lost. I was thinking the same that the threads are re-used so very few metrics
should be lost, and also from within the {{FSNamesystemLock}} there wouldn't be
any easy way to detect that the thread is exiting to flush its metrics. I think
this is worth documenting though; I've added some discussion on the
{{FSNamesystemLock}} Javadoc.
2. Good point. I've added some guidance about a good default in
{{hdfs-default.xml}}.
3. This seems like a pretty interesting idea. It could be pretty simple /
little implementation effort, but I think there is some question of if it will
actually be helpful. I can code something up for this patch or leave it... Let
me know what you think.
1. Right. That {{yield()}} is only called as part of the {{contentSummary}}
operation, just quickly releasing and reacquiring the lock which
{{contentSummary}} originally held. The {{yield}} could make a single
{{contentSummary}} appear as multiple operations instead of a single one, but
the total time should still be accurate.
2. For any operation that already had an audit log name I kept that for
consistency. Agreed that it doesn't make sense to change here.
3. Thanks, fixed.
4. Yes, agreed. Not sure why I originally went with {{checkLease}}. I've
changed it to {{leaseManager}} to be consistent with the name of the class.
5. Agreed, changed. Good catch.
Attaching v003 patch with changes discussed.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586511#comment-15586511
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 830 unchanged - 0 fixed = 831 total (was 830) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 48s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12833997/HDFS-10872.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 054dc15365c5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b733a6f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17205/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17205/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17205/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17205/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add MutableRate me
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587033#comment-15587033
]
Zhe Zhang commented on HDFS-10872:
--
Thanks Erik! The v3 patch LGTM. +1 pending fixing the long line checkstyle
issue.
{code}
* {@link
org.apache.hadoop.hdfs.DFSConfigKeys#DFS_NAMENODE_LOCK_METRIC_AGG_INTERVAL_MS_KEY}
{code}
I think we should leave the try-and-backoff as a potential optimization.
I will hold off committing the patch until EOD Thursday to collect more
feedbacks. [~kihwal] Do you think the patch addresses your [use cases |
https://issues.apache.org/jira/browse/HDFS-10475?focusedCommentId=15490522&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15490522]?
[~andrew.wang] Do you think the aggregation method addresses your concern on
performance (see benchmark above)? [~liuml07] Any comments on the form of
output?
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587087#comment-15587087
]
Andrew Wang commented on HDFS-10872:
Thanks for working on this folks, and sorry for coming to this review late. I
have a few comments looking at this patch:
* I'd prefer we use try/finally to do the lock/unlocks in aggregateMetrics for
safety.
* We're doing the thread-local increment and aggregate outside of the metrics
system right now. Did we look into making the metrics classes themselves
multi-threading friendly? Then we could more easily reuse this code in other
places.
* Related, the aggregation right now is done periodically on increment. I think
what we'd really like is aggregate-on-read. The metrics classes support this
via {{setChanged}} and {{snapshot}}. Otherwise, occasionally one operation is
getting hit with the penalty of aggregating all the metrics, and possibly for
no reason if it's not being read.
I might have mentioned this before, but the
[LongAdder|http://gee.cs.oswego.edu/cgi-bin/viewcvs.cgi/jsr166/src/jsr166e/LongAdder.java?view=co]
code in JDK8 is basically what I'm thinking of, and the performance is quite
good. The license is friendly, so we could copy-paste this into Hadoop for the
branch-2's, and use the built-in JDK8 version for trunk.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch,
> HDFS-10872.004.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587290#comment-15587290
]
Erik Krogen commented on HDFS-10872:
[~andrew.wang], thanks for the comment.
I actually did not think to modify the metrics classes rather than working
around their current limitations but this is definitely good to consider. We
are specifically interested in the MutableRate metric here, which is currently
{{synchronized}} since it requires updating of multiple values atomically. For
something like MutableCounter a LongAdder would be great, but I am wondering if
we can make it apply here? I particularly note the Javadoc which says, of
resets,
bq. Because this method is intrinsically racy, it should only be used when it
is known that no threads are concurrently updating.
But we would need the ability to do resets for MutableRate. I suppose we could
potentially read lock around a normal increment of a LongAdder and write lock
upon snapshotting; or, probably more performant, use an AtomicReference to
store the LongAdder and create a new one when we snapshot rather than
attempting to reset. Any thoughts on that?
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch,
> HDFS-10872.004.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15587300#comment-15587300
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
8s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 830 unchanged - 0 fixed = 831 total (was 830) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation |
| | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834065/HDFS-10872.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux e0ae13c476f1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c62ae71 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17210/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17210/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Buil
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589506#comment-15589506
]
Zhe Zhang commented on HDFS-10872:
--
Thanks for the discussion Andrew and Erik.
I agree that moving the multi-threading logic to Metrics classes is a good
idea. I'm actually pretty curious how that will improve the performance of
existing RPC metrics collection; it seems pretty expensive. We should probably
do that as a prerequisite and then rebase this patch.
Using {{LongAdder}} is an interesting idea. [~andrew.wang] This is actually the
first time I learn about it :) Are we aware of any production usage of this
library? I took a look at the test coverage; it seems a little weak considering
the complexity of the main logic. We can probably consider adding some more
unit tests in Hadoop to cover cases we are most interested in.
I expect that a few minutes after the NN starts, the set of operations (keys)
in the {{opHoldtimeMetrics}} map is stablized. Afterwards, each entry is
essentially a counter to be incremented. So on the high level I think we should
be able to make it work with {{LongAdder}}. Agreed with [~xkrogen] that
snapshotting/resetting is probably the trickiest part. Creating a new
{{LongAdder}} in {{snapshot}} sounds reseaonable to me.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch,
> HDFS-10872.004.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589917#comment-15589917 ] Andrew Wang commented on HDFS-10872: Thanks for the replies, Erik, Zhe, inline: bq. Are we aware of any production usage of this library? The popular codahale/Dropwizard metrics library pulls in LongAdder in the manner I suggested, so it might be easiest to add that as a dependency. I'm pretty sure we already have it floating around in Hadoop somewhere. The quality of JDK code is also well regarded. bq. For something like MutableCounter a LongAdder would be great, but I am wondering if we can make it apply here? Looking at [EWMA in codahale|https://github.com/dropwizard/metrics/blob/3.2-development/metrics-core/src/main/java/com/codahale/metrics/EWMA.java], they reset a LongAdder without additional synchronization with updates. Metrics don't need to be super precise, so as long as there isn't a disastrous concurrent failure mode, this is okay. Unless we really need the additional bits of MutableRate, the simplest thing might be to wrap the Dropwizard metrics classes like Histogram and EWMA to fit in the Hadoop metrics system. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595745#comment-15595745
]
Erik Krogen commented on HDFS-10872:
[~andrew.wang] I did some profiling and it seems that the performance hit to
keeping a consistent view and using an atomic reference is pretty negligible.
Benchmark results below, running each benchmark for 20 seconds with 100
competing reader threads incrementing the counter as quickly as they can and a
varying snapshot interval:
{code}
Benchmark (metricType) (snapshotIntervalMs) Mode
Cnt Score Error Units
MetricBenchmark.increment Synchronized 100 avgt
10 10802.943 ± 366.821 ns/op
MetricBenchmark.increment Synchronized 1000 avgt
10 8991.085 ± 224.738 ns/op
MetricBenchmark.increment Synchronized 5000 avgt
10 9780.644 ± 258.482 ns/op
MetricBenchmark.increment Synchronized 1 avgt
10 8864.450 ± 239.349 ns/op
MetricBenchmark.increment RWLockAdder 100 avgt
10 31546.824 ± 203.405 ns/op
MetricBenchmark.increment RWLockAdder 1000 avgt
10 31350.341 ± 156.414 ns/op
MetricBenchmark.increment RWLockAdder 5000 avgt
10 8.370 ± 234.421 ns/op
MetricBenchmark.increment RWLockAdder 1 avgt
10 31057.551 ± 211.002 ns/op
MetricBenchmark.increment AtomicReferenceAdder 100 avgt
10172.594 ± 1.478 ns/op
MetricBenchmark.increment AtomicReferenceAdder 1000 avgt
10171.760 ± 0.647 ns/op
MetricBenchmark.increment AtomicReferenceAdder 5000 avgt
10172.942 ± 2.991 ns/op
MetricBenchmark.increment AtomicReferenceAdder 1 avgt
10186.659 ± 23.906 ns/op
MetricBenchmark.increment UnprotectedAdder 100 avgt
10202.249 ± 0.371 ns/op
MetricBenchmark.increment UnprotectedAdder 1000 avgt
10166.536 ± 0.273 ns/op
MetricBenchmark.increment UnprotectedAdder 5000 avgt
10179.672 ± 0.335 ns/op
MetricBenchmark.increment UnprotectedAdder 1 avgt
10168.062 ± 2.149 ns/op
{code}
Synchronized is basically the current approach (a {{synchronized}} method),
RWLockAdder uses a {{ReadWriteLock}} to protect a {{LongAdder}},
AtomicReferenceAdder uses an {{AtomicReference}} to protect a {{LongAdder}},
and UnprotectedAdder simply uses a {{LongAdder}} with an unprotected
{{sumThenReset}}. It seems based off of this that using an AtomicReference to
protect a {{LongAdder}} (or, an object containing a small number of LongAdders
for e.g. {{MutableRate}} which needs to maintain multiple counters) would
provide a consistent snapshot and be much more performant than our current
solution.
Thoughts?
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch,
> HDFS-10872.004.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595792#comment-15595792 ] Zhe Zhang commented on HDFS-10872: -- Thanks for the benchmark [~xkrogen]! Results look very interesting. I think it's time to move the discussion to a separate metrics JIRA. Just created HADOOP-13747. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595794#comment-15595794 ] Zhe Zhang commented on HDFS-10872: -- Thanks for the benchmark [~xkrogen]! Results look very interesting. I think it's time to move the discussion to a separate metrics JIRA. Just created HADOOP-13747. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15595927#comment-15595927 ] Andrew Wang commented on HDFS-10872: This looks great, thanks Erik! I was a bit worried since LongAdder needs to expand the hashtable as it warms up, but the hit looks marginal. Let's pursue this further in HADOOP-13747. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15652905#comment-15652905
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 49s{color} | {color:orange} root: The patch generated 2 new + 862 unchanged
- 0 fixed = 864 total (was 862) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
7s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 41s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.server.namenode.TestMetaSave |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838271/HDFS-10872.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux ec833f598ec1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 71adf44 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17494/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17494/artifact/patchprocess/patch-unit-hadoop
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654899#comment-15654899
]
Zhe Zhang commented on HDFS-10872:
--
Thanks Erik for the patch! LGTM overall. I think the added
{{newMutableRatesWithAggregation}} constructor is reasonable. Some minors:
# Below change doesn't seem necessary
{code}
- private final MetricsRegistry registry = new MetricsRegistry("FSNamesystem");
+ private final MetricsRegistry registry =
+ new MetricsRegistry("FSNamesystem").setContext("dfs");
{code}
# We could have the @link have its own line, right now the link is broken:
{code}
* If metrics are enabled by setting {@link org.apache.hadoop.hdfs.DFSConfigKeys
* #DFS_NAMENODE_LOCK_DETAILED_METRICS_KEY} to be true, metrics will be emitted
{code}
# Follow-on: maybe we can consider consolidating the code in readUnlock and
writeUnlock
[~andrew.wang] Any further comments? Thanks.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch,
> HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654905#comment-15654905
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
15s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 7m
7s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
56s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m
39s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 39s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 45s{color} | {color:orange} root: The patch generated 1 new + 862 unchanged
- 0 fixed = 863 total (was 862) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
21s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 31s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838401/HDFS-10872.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 4cd2b8c865a8 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 89354f0 |
| Default Java | 1.8.0_101 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/branch-compile-root.txt
|
| findbugs | v3.0.0 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/17509/artifact/patchprocess/patch-compile-root.txt
|
| javac |
https://builds.apache.org/j
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15654931#comment-15654931 ] Erik Krogen commented on HDFS-10872: [~zhz], thanks for the review. (1) Yes, you are right, that is unnecessary. (2) Fixed. I'm surprised my IDE didn't warn me of this. (3) Agreed that this could be a good follow-up, there is a lot of duplicated logic. Could be good to do something to consolidate with HADOOP-13702 as well. Attaching v007 patch. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655243#comment-15655243
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
58s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 7m
21s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 6m
14s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 14s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 49s{color} | {color:orange} root: The patch generated 2 new + 862 unchanged
- 0 fixed = 864 total (was 862) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
2s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}115m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestMissingBlocksAlert |
| | hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.tools.TestDFSAdmin |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838420/HDFS-10872.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux cfea7118d697 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 89354f0 |
| Default Java | 1.8.0_101 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/17513/artifact/patchprocess/branch-compile-root.txt
|
| findbugs | v3.0.0 |
| c
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15655560#comment-15655560
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 7m
35s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m
59s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 59s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 44s{color} | {color:orange} root: The patch generated 2 new + 862 unchanged
- 0 fixed = 864 total (was 862) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
44s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 20s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838420/HDFS-10872.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 8c888dce6de0 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 89354f0 |
| Default Java | 1.8.0_101 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/branch-compile-root.txt
|
| findbugs | v3.0.0 |
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/17517/artifact/patchprocess/patch-compile-root.txt
|
| javac |
https://builds.apache.org/j
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658222#comment-15658222
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 44s{color} | {color:orange} root: The patch generated 2 new + 862 unchanged
- 0 fixed = 864 total (was 862) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
29s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 10s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}117m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.TestRenameWhileOpen |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838602/HDFS-10872.009.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 621f3278cf6d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5c61ad2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17533/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/1
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658252#comment-15658252 ] Erik Krogen commented on HDFS-10872: TestDFSAdmin is a known issue tracked in HDFS-11122. TestRenameWhileOpen/TestDelegationTokenFetcher pass locally and seem unrelated. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658296#comment-15658296 ] Zhe Zhang commented on HDFS-10872: -- Thanks [~wangda] for helping fix the npm error. Thanks Erik for verifying pre-commit. +1 on the patch, will work on trunk~branch-2.7 commit now. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658299#comment-15658299 ] Mingliang Liu commented on HDFS-10872: -- +1 > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658342#comment-15658342 ] Zhe Zhang commented on HDFS-10872: -- Sorry, just realized an issue with the names of the metrics, can we more explicitly specify those metrics are for locks? > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658355#comment-15658355
]
Mingliang Liu commented on HDFS-10872:
--
That's a good point. We can add prefix (e.g. FSNLock) in
{{FSNamesystemLock#addMetric()}}.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch,
> HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch,
> HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658378#comment-15658378
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 23 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
22s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 9m
10s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 4m
34s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui hadoop-cloud-storage-project
hadoop-cloud-storage-project/hadoop-cloud-storage {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
41s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m
57s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 5m 57s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 57s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 38s{color} | {color:orange} root: The patch generated 39 new + 2837
unchanged - 11 fixed = 2876 total (was 2848) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 10 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red}
patch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
no findbugs output file
(hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/target/findbugsXml.xml)
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 4m
23s{color} | {color:red} root generated 11198 new + 0 unchanged - 0 fixed =
11198 total (was 0) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}140m 5s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 9 ASF License warnings. {color} |
| {color:bla
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15658608#comment-15658608 ] Mingliang Liu commented on HDFS-10872: -- still +1 Thanks for updating the code. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch, > HDFS-10872.010.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15659125#comment-15659125 ] Mingliang Liu commented on HDFS-10872: -- Thanks for taking care of this. I usually don't bump the patch version number; the JIRA will show old patch of same version (well, name) as gray so it's not confusing anyway. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch, > HDFS-10872.010.patch, HDFS-10872.011.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15659343#comment-15659343
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m
5s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 5s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 45s{color} | {color:orange} root: The patch generated 2 new + 862 unchanged
- 0 fixed = 864 total (was 862) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
14s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 16s{color}
| {color:red} hadoop-yarn-ui in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15659347#comment-15659347
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m
17s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 17s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 40s{color} | {color:orange} root: The patch generated 2 new + 862 unchanged
- 0 fixed = 864 total (was 862) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 11s{color}
| {color:red} hadoop-yarn-ui in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838646/HDFS-10
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664399#comment-15664399 ] Zhe Zhang commented on HDFS-10872: -- Thanks for the tip Mingliang :) > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch, > HDFS-10872.010.patch, HDFS-10872.011.patch, HDFS-10872.012.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664641#comment-15664641
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 43s{color} | {color:orange} root: The patch generated 2 new + 862 unchanged
- 0 fixed = 864 total (was 862) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
23s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838788/HDFS-10872.012.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 5c6ac6a2acba 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6efb8c9 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17552/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17552/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.t
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664718#comment-15664718 ] Zhe Zhang commented on HDFS-10872: -- Thanks Erik. +1 on v12 patch. I did another check of the metrics from local cluster, they look good. I just pushed to trunk, working on branch backports. Thanks for the great work! > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch, > HDFS-10872.010.patch, HDFS-10872.011.patch, HDFS-10872.012.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664725#comment-15664725
]
Zhe Zhang commented on HDFS-10872:
--
Actually {{FSNamesystem}} has many conflicts -- though most of them look pretty
mechanical. [~xkrogen] Do you mind creating a branch-2 patch? I guess we should
also wait for branch-2 Jenkins precommit run.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872.000.patch,
> HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch,
> HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch,
> HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch,
> HDFS-10872.010.patch, HDFS-10872.011.patch, HDFS-10872.012.patch, jmx-output
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664745#comment-15664745 ] Hudson commented on HDFS-10872: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10832 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10832/]) HDFS-10872. Add MutableRate metrics for FSNamesystemLock operations. (zhz: rev ff0b99eafeda035ebe0dc82cfe689808047a8893) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MetricsRegistry.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystemLock.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemLock.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableMetricsFactory.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872.000.patch, > HDFS-10872.001.patch, HDFS-10872.002.patch, HDFS-10872.003.patch, > HDFS-10872.004.patch, HDFS-10872.005.patch, HDFS-10872.006.patch, > HDFS-10872.007.patch, HDFS-10872.008.patch, HDFS-10872.009.patch, > HDFS-10872.010.patch, HDFS-10872.011.patch, HDFS-10872.012.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664853#comment-15664853
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
2s{color} | {color:red} Docker failed to build yetus/hadoop:b59b8b7. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-10872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12838829/HDFS-10872-branch-2.012.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17556/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add MutableRate metrics for FSNamesystemLock operations
> ---
>
> Key: HDFS-10872
> URL: https://issues.apache.org/jira/browse/HDFS-10872
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Erik Krogen
>Assignee: Erik Krogen
> Attachments: FSLockPerf.java, HDFS-10872-branch-2.012.patch,
> HDFS-10872.000.patch, HDFS-10872.001.patch, HDFS-10872.002.patch,
> HDFS-10872.003.patch, HDFS-10872.004.patch, HDFS-10872.005.patch,
> HDFS-10872.006.patch, HDFS-10872.007.patch, HDFS-10872.008.patch,
> HDFS-10872.009.patch, HDFS-10872.010.patch, HDFS-10872.011.patch,
> HDFS-10872.012.patch, jmx-output
>
>
> Add metrics for FSNamesystemLock operations to see, overall, how long each
> operation is holding the lock for. Use MutableRate metrics for now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15664936#comment-15664936 ] Zhe Zhang commented on HDFS-10872: -- Thanks Erik. Local cluster on branch-2 patch looks good. I just retriggered Jenkins. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872-branch-2.012.patch, > HDFS-10872.000.patch, HDFS-10872.001.patch, HDFS-10872.002.patch, > HDFS-10872.003.patch, HDFS-10872.004.patch, HDFS-10872.005.patch, > HDFS-10872.006.patch, HDFS-10872.007.patch, HDFS-10872.008.patch, > HDFS-10872.009.patch, HDFS-10872.010.patch, HDFS-10872.011.patch, > HDFS-10872.012.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665459#comment-15665459
]
Hadoop QA commented on HDFS-10872:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
31s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
57s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
57s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
56s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
56s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 41s{color} | {color:orange} root: The patch generated 4 new + 859 unchanged
- 0 fixed = 863 total (was 859) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
47s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
6s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 67m
47s{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_111.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}223m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeMultip
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[ https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665527#comment-15665527 ] Zhe Zhang commented on HDFS-10872: -- Thanks Erik! I manually verified the 2 checkstyle issues. I just committed to branch-2. > Add MutableRate metrics for FSNamesystemLock operations > --- > > Key: HDFS-10872 > URL: https://issues.apache.org/jira/browse/HDFS-10872 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Erik Krogen >Assignee: Erik Krogen > Attachments: FSLockPerf.java, HDFS-10872-branch-2.012.patch, > HDFS-10872-branch-2.013.patch, HDFS-10872.000.patch, HDFS-10872.001.patch, > HDFS-10872.002.patch, HDFS-10872.003.patch, HDFS-10872.004.patch, > HDFS-10872.005.patch, HDFS-10872.006.patch, HDFS-10872.007.patch, > HDFS-10872.008.patch, HDFS-10872.009.patch, HDFS-10872.010.patch, > HDFS-10872.011.patch, HDFS-10872.012.patch, jmx-output > > > Add metrics for FSNamesystemLock operations to see, overall, how long each > operation is holding the lock for. Use MutableRate metrics for now. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15666024#comment-15666024
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
2s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
13s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
54s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
2s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
18s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
47s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 55s{color} | {color:orange} root: The patch generated 2 new + 859 unchanged
- 0 fixed = 861 total (was 859) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
59s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
21s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_111. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 34s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}245m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_101 Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| JDK v1.7.0_111 Failed j
[jira] [Commented] (HDFS-10872) Add MutableRate metrics for FSNamesystemLock operations
[
https://issues.apache.org/jira/browse/HDFS-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15666091#comment-15666091
]
Hadoop QA commented on HDFS-10872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m
50s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
14s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
20s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s{color} | {color:green} branch-2.7 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
49s{color} | {color:red} hadoop-common-project/hadoop-common in branch-2.7 has
3 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
53s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
22s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
7s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 36s{color} | {color:orange} root: The patch generated 5 new + 1115 unchanged
- 1 fixed = 1120 total (was 1116) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 7079 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 3m
34s{color} | {color:red} The patch 185 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} the patch passed with JDK v1.8.0_111 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
56s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 14s{color}
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_111. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 32s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_111. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
21s{color} | {color:red} T
