[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038342#comment-15038342
]
Konstantin Shvachko commented on HDFS-7847:
---
I created HDFS-9503 to replace the {{-namenode}} option with {{-fs}}.
Although I still don't understand the use case for a stand alone NameNode
benchmark.
> Modify NNThroughputBenchmark to be able to operate on a remote NameNode
> ---
>
> Key: HDFS-7847
> URL: https://issues.apache.org/jira/browse/HDFS-7847
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Colin Patrick McCabe
>Assignee: Charles Lamb
> Fix For: 2.8.0
>
> Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch,
> HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch,
> HDFS-7847.005.patch, make_blocks.tar.gz
>
>
> Modify NNThroughputBenchmark to be able to operate on a NN that is not in
> process. A followon Jira will modify it some more to allow quantifying native
> and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014253#comment-15014253
]
Konstantin Shvachko commented on HDFS-7847:
---
Yes, {{NNThroughputBenchmark}} is a benchmark. Glad we established this fact.
Now, could you (or Charles, or anybody) please explain how you plan to use it
as _a benchmark with a standalone NN_. I mean things like
- what are you measuring
- how you set up the nodes
- are there any particular RPC improvements or alternative implementations you
are comparing
Thought I was asking a simple question, but it is getting confusing.
> Modify NNThroughputBenchmark to be able to operate on a remote NameNode
> ---
>
> Key: HDFS-7847
> URL: https://issues.apache.org/jira/browse/HDFS-7847
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Colin Patrick McCabe
>Assignee: Charles Lamb
> Fix For: 2.8.0
>
> Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch,
> HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch,
> HDFS-7847.005.patch, make_blocks.tar.gz
>
>
> Modify NNThroughputBenchmark to be able to operate on a NN that is not in
> process. A followon Jira will modify it some more to allow quantifying native
> and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015171#comment-15015171 ] Colin Patrick McCabe commented on HDFS-7847: bq. what are you measuring NNThroughput, measured in ops per second. bq. how you set up the nodes Any way you like, depending on what you are benchmarking. There is no special setup needed. bq. are there any particular RPC improvements or alternative implementations you are comparing Somewhat recently, I did some benchmarks to figure out how to tune the NN to get the most ops per second. Actually on my particular hardware, fsync turned out to be a bottleneck, since hard disk write caching was turned off. In general, fsync is not a bottleneck, but it was in my specific case. More generally, we should pay attention to benchmarks like these to make sure our performance doesn't drop. > Modify NNThroughputBenchmark to be able to operate on a remote NameNode > --- > > Key: HDFS-7847 > URL: https://issues.apache.org/jira/browse/HDFS-7847 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Colin Patrick McCabe >Assignee: Charles Lamb > Fix For: 2.8.0 > > Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, > HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, > HDFS-7847.005.patch, make_blocks.tar.gz > > > Modify NNThroughputBenchmark to be able to operate on a NN that is not in > process. A followon Jira will modify it some more to allow quantifying native > and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012903#comment-15012903
]
Colin Patrick McCabe commented on HDFS-7847:
bq. If understand this correctly, you are not targeting to build a benchmark,
but rather unit tests of NameNode RPC. Isn't there a bunch of those in HDFS
source tree already? Just trying to understand what is different.
Excuse me, I mis-spoke. {{NNThroughputBenchmark}} is a benchmark, not a unit
test.
bq. Have you looked at org.apache.hadoop.fs.loadGenerator.LoadGenerator. Would
its functionality suffice the testing needs you have in mind?
{{LoadGenerator}} is certainly much more similar to {{NNThroughputBenchmark}}
than any of the other things you mentioned. However, it performs a mix of only
list, read, and write operations. {{NNThroughputBenchmark}} performs a wider
range of operations. In general, I would like to avoid read and write
operations in my testing of NN RPC performance, since those operations involve
the DataNode heavily. I think it's good to have multiple different benchmarks
testing different things.
> Modify NNThroughputBenchmark to be able to operate on a remote NameNode
> ---
>
> Key: HDFS-7847
> URL: https://issues.apache.org/jira/browse/HDFS-7847
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Colin Patrick McCabe
>Assignee: Charles Lamb
> Fix For: 2.8.0
>
> Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch,
> HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch,
> HDFS-7847.005.patch, make_blocks.tar.gz
>
>
> Modify NNThroughputBenchmark to be able to operate on a NN that is not in
> process. A followon Jira will modify it some more to allow quantifying native
> and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009837#comment-15009837
]
Konstantin Shvachko commented on HDFS-7847:
---
??The motivation was to have a simple unit test of raw NameNode RPC throughput
in the HDFS source code.??
If understand this correctly, you are not targeting to build a benchmark, but
rather unit tests of NameNode RPC.
Isn't there a bunch of those in HDFS source tree already? Just trying to
understand what is different.
??DFSIO is not a test of pure NN RPC performance, since it is a MapReduce job??
It can be used to test NameNode's performance, because the MR overhead is
subtracted. It is in MapReduce though, true.
??SLive is an external project??
SLive is in the same place as DFSIO. External to what?
Have you looked at
[{{org.apache.hadoop.fs.loadGenerator.LoadGenerator}}|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/loadGenerator/LoadGenerator.java].
Would its functionality suffice the testing needs you have in mind?
> Modify NNThroughputBenchmark to be able to operate on a remote NameNode
> ---
>
> Key: HDFS-7847
> URL: https://issues.apache.org/jira/browse/HDFS-7847
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Colin Patrick McCabe
>Assignee: Charles Lamb
> Fix For: 2.8.0
>
> Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch,
> HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch,
> HDFS-7847.005.patch, make_blocks.tar.gz
>
>
> Modify NNThroughputBenchmark to be able to operate on a NN that is not in
> process. A followon Jira will modify it some more to allow quantifying native
> and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009534#comment-15009534
]
Konstantin Shvachko commented on HDFS-7847:
---
Shouldn't we be specifying the namenode using the standard generic option
{{-fs}}?
Also {{NNThroughputBenchmark}} was intended to benchmark NameNode's performance
without RPC overhead.
There are other tools which can benchmark a real NameNode, like
- DFSIO,
- SLive,
- synthetic load generator or
I guess I did not fully understand your motivation here?
> Modify NNThroughputBenchmark to be able to operate on a remote NameNode
> ---
>
> Key: HDFS-7847
> URL: https://issues.apache.org/jira/browse/HDFS-7847
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Colin Patrick McCabe
>Assignee: Charles Lamb
> Fix For: 2.8.0
>
> Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch,
> HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch,
> HDFS-7847.005.patch, make_blocks.tar.gz
>
>
> Modify NNThroughputBenchmark to be able to operate on a NN that is not in
> process. A followon Jira will modify it some more to allow quantifying native
> and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15009722#comment-15009722 ] Colin Patrick McCabe commented on HDFS-7847: bq. Shouldn't we be specifying the namenode using the standard generic option -fs? That makes sense to me. Would you like to post a follow-on JIRA? bq. I guess I did not fully understand your motivation here? The motivation was to have a simple unit test of raw NameNode RPC throughput in the HDFS source code. DFSIO is not a test of pure NN RPC performance, since it is a MapReduce job with all the attendant overheads. SLive is an external project not in the HDFS source tree. If you want to benchmark the NameNode's performance without RPC overhead, you can still do that with NNThroughputBenchmark, by using the right command-line options. > Modify NNThroughputBenchmark to be able to operate on a remote NameNode > --- > > Key: HDFS-7847 > URL: https://issues.apache.org/jira/browse/HDFS-7847 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: Colin Patrick McCabe >Assignee: Charles Lamb > Fix For: 2.8.0 > > Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, > HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, > HDFS-7847.005.patch, make_blocks.tar.gz > > > Modify NNThroughputBenchmark to be able to operate on a NN that is not in > process. A followon Jira will modify it some more to allow quantifying native > and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530500#comment-14530500 ] Hudson commented on HDFS-7847: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #919 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/919/]) HDFS-7847. Modify NNThroughputBenchmark to be able to operate on a remote NameNode (Charles Lamb via Colin P. McCabe) (cmccabe: rev ffce9a3413277a69444fcb890460c885de56db69) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Fix For: 2.8.0 Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, HDFS-7847.005.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530455#comment-14530455 ] Hudson commented on HDFS-7847: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #186 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/186/]) HDFS-7847. Modify NNThroughputBenchmark to be able to operate on a remote NameNode (Charles Lamb via Colin P. McCabe) (cmccabe: rev ffce9a3413277a69444fcb890460c885de56db69) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Fix For: 2.8.0 Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, HDFS-7847.005.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530877#comment-14530877 ] Hudson commented on HDFS-7847: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2135 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2135/]) HDFS-7847. Modify NNThroughputBenchmark to be able to operate on a remote NameNode (Charles Lamb via Colin P. McCabe) (cmccabe: rev ffce9a3413277a69444fcb890460c885de56db69) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Fix For: 2.8.0 Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, HDFS-7847.005.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530904#comment-14530904 ] Hudson commented on HDFS-7847: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #186 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/186/]) HDFS-7847. Modify NNThroughputBenchmark to be able to operate on a remote NameNode (Charles Lamb via Colin P. McCabe) (cmccabe: rev ffce9a3413277a69444fcb890460c885de56db69) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Fix For: 2.8.0 Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, HDFS-7847.005.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530750#comment-14530750 ] Hudson commented on HDFS-7847: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #176 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/176/]) HDFS-7847. Modify NNThroughputBenchmark to be able to operate on a remote NameNode (Charles Lamb via Colin P. McCabe) (cmccabe: rev ffce9a3413277a69444fcb890460c885de56db69) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Fix For: 2.8.0 Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, HDFS-7847.005.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530729#comment-14530729 ] Hudson commented on HDFS-7847: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2117 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2117/]) HDFS-7847. Modify NNThroughputBenchmark to be able to operate on a remote NameNode (Charles Lamb via Colin P. McCabe) (cmccabe: rev ffce9a3413277a69444fcb890460c885de56db69) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Fix For: 2.8.0 Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, HDFS-7847.005.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14528403#comment-14528403
]
Hadoop QA commented on HDFS-7847:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 13s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 15s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 0s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 1m 20s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 166m 19s | Tests failed in hadoop-hdfs. |
| | | 188m 5s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12730467/HDFS-7847.005.patch |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 318081c |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/10809/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/10809/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10809/console |
This message was automatically generated.
Modify NNThroughputBenchmark to be able to operate on a remote NameNode
---
Key: HDFS-7847
URL: https://issues.apache.org/jira/browse/HDFS-7847
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Charles Lamb
Fix For: HDFS-7836
Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch,
HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch,
HDFS-7847.005.patch, make_blocks.tar.gz
Modify NNThroughputBenchmark to be able to operate on a NN that is not in
process. A followon Jira will modify it some more to allow quantifying native
and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14528998#comment-14528998 ] Colin Patrick McCabe commented on HDFS-7847: +1. Thanks, [~clamb]. Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Fix For: HDFS-7836 Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, HDFS-7847.005.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529036#comment-14529036 ] Hudson commented on HDFS-7847: -- FAILURE: Integrated in Hadoop-trunk-Commit #7739 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7739/]) HDFS-7847. Modify NNThroughputBenchmark to be able to operate on a remote NameNode (Charles Lamb via Colin P. McCabe) (cmccabe: rev ffce9a3413277a69444fcb890460c885de56db69) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Fix For: 2.8.0 Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, HDFS-7847.005.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527525#comment-14527525 ] Colin Patrick McCabe commented on HDFS-7847: [~clamb], can you rebase this on trunk? Looks like it's gotten stale Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Fix For: HDFS-7836 Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch, HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527502#comment-14527502
]
Hadoop QA commented on HDFS-7847:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12728988/HDFS-7847.004.patch |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 0d6aa5d |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10795/console |
This message was automatically generated.
Modify NNThroughputBenchmark to be able to operate on a remote NameNode
---
Key: HDFS-7847
URL: https://issues.apache.org/jira/browse/HDFS-7847
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Charles Lamb
Fix For: HDFS-7836
Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch,
HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch,
make_blocks.tar.gz
Modify NNThroughputBenchmark to be able to operate on a NN that is not in
process. A followon Jira will modify it some more to allow quantifying native
and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527391#comment-14527391
]
Hadoop QA commented on HDFS-7847:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12728988/HDFS-7847.004.patch |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / bf70c5a |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/10794/console |
This message was automatically generated.
Modify NNThroughputBenchmark to be able to operate on a remote NameNode
---
Key: HDFS-7847
URL: https://issues.apache.org/jira/browse/HDFS-7847
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Charles Lamb
Fix For: HDFS-7836
Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch,
HDFS-7847.002.patch, HDFS-7847.003.patch, HDFS-7847.004.patch,
make_blocks.tar.gz
Modify NNThroughputBenchmark to be able to operate on a NN that is not in
process. A followon Jira will modify it some more to allow quantifying native
and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14365842#comment-14365842
]
Colin Patrick McCabe commented on HDFS-7847:
{{getNamenodeProtocolProxy}}, {{getRefreshUserMappingsProtocolProxy}}: these
don't belong in DFSClient. They've got nothing to do with the client, only
with HDFS unit tests. Put this stuff someplace like {{DFSTestUtil.java}}
+1 once that's fixed. Thanks, [~clamb].
Modify NNThroughputBenchmark to be able to operate on a remote NameNode
---
Key: HDFS-7847
URL: https://issues.apache.org/jira/browse/HDFS-7847
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7836
Reporter: Colin Patrick McCabe
Assignee: Charles Lamb
Attachments: HDFS-7847.000.patch, HDFS-7847.001.patch,
HDFS-7847.002.patch, make_blocks.tar.gz
Modify NNThroughputBenchmark to be able to operate on a NN that is not in
process. A followon Jira will modify it some more to allow quantifying native
and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[
https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14353892#comment-14353892
]
Colin Patrick McCabe commented on HDFS-7847:
DFSClient.java: this change adds three new fields to DFSClient. But they only
seem to be used by unit tests. It seems like we should just put these inside
the unit test(s) that are using these-- if necessary, by adding a helper
method. There's no reason to add more fields to DFSClient. Also remember that
when using FileContext, we create new DFSClients all the time.
{{this.datanodeProtocolProxy = NameNodeProxies.createProxy...}}
It seems kind of odd to have {{NameNodeProxies#createProxy}} create a proxy to
the datanode. In general, when you see NameNodeProxies I think proxies used
by the NameNode and this doesn't fit with that. Can you give a little more
context about why this is a good idea (as opposed to just having some custom
code in the unit test or in a unit test util class that creates a proxy)
{code}
121 [-remoteNamenode namenode URI]\n +
122 If using -remoteNamenode, set the namenode's +
123 dfs.namenode.fs-limits.min-block-size to 16.;
{code}
Of course the NameNode may or may not be remote here. It seems like
{{\-\-nnuri}} or just {{\-\-namenode}} or something like that would be more
descriptive.
{code}
378 final int remoteNNIndex = args.indexOf(-remoteNamenode);
379 if (remoteNNIndex = 0) {
380 if (args.size() = remoteNNIndex + 1) {
381 printUsage();
382 }
383 remoteNamenode = args.get(remoteNNIndex + 1);
384 args.remove(remoteNNIndex + 1);
385 args.remove(remoteNNIndex);
386 }
{code}
Instead of this boilerplate, just use {{StringUtils#popOptionWithArgument}}
{code}
- replication, BLOCK_SIZE, null);
+ replication, BLOCK_SIZE, CryptoProtocolVersion.supported());
{code}
This fix is a little bit separate, right? I suppose we can do it in this JIRA,
though.
Modify NNThroughputBenchmark to be able to operate on a remote NameNode
---
Key: HDFS-7847
URL: https://issues.apache.org/jira/browse/HDFS-7847
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7836
Reporter: Colin Patrick McCabe
Assignee: Charles Lamb
Attachments: HDFS-7847.000.patch, make_blocks.tar.gz
Modify NNThroughputBenchmark to be able to operate on a NN that is not in
process. A followon Jira will modify it some more to allow quantifying native
and java heap sizes, and some latency numbers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354259#comment-14354259 ] stack commented on HDFS-7847: - Mighty [~clamb] I forgot about that proxy stuff. It was never pretty So idea is that you add proxies for protocols' currently hardwired? That sounds needed if we want to fake out DFSClient that there are thousands of DNs with millions of blocks each. How comes I don't see your added code replacing current hard-wiring of DatanodeProtocol, etc? On NNThroughputBenchmark, I think you should just steal the good bits and leave the rest behind you. I say this because I don't think you will be able to put up a load on a NN in a unit test that emulates thousands of nodes making mega-reports about millions upon millions of blocks while simulating thousands of clients. I'd think you'll want to stand up a NN and then batter it for hours on end from many processes distributed over a cluster. Just a suggestion. Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-7836 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Attachments: HDFS-7847.000.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7847) Modify NNThroughputBenchmark to be able to operate on a remote NameNode
[ https://issues.apache.org/jira/browse/HDFS-7847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354332#comment-14354332 ] stack commented on HDFS-7847: - (Offlist) [~clamb] set me right, that this patch actually allows you go bang on remote NN... so scratch my above remark (I don't seem to be able to edit it). Modify NNThroughputBenchmark to be able to operate on a remote NameNode --- Key: HDFS-7847 URL: https://issues.apache.org/jira/browse/HDFS-7847 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-7836 Reporter: Colin Patrick McCabe Assignee: Charles Lamb Attachments: HDFS-7847.000.patch, make_blocks.tar.gz Modify NNThroughputBenchmark to be able to operate on a NN that is not in process. A followon Jira will modify it some more to allow quantifying native and java heap sizes, and some latency numbers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
