[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172507#comment-14172507 ] Colin Patrick McCabe commented on HDFS-7055: bq. Yongjun did not work or even comment on HDFS-7169. Typo? Ah. I was confused because originally he did plan to fix it in HDFS-7194, but then we refocused the JIRA on just one findbugs issue. And then you fixed it in the other JIRA. > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171629#comment-14171629 ] Tsz Wo Nicholas Sze commented on HDFS-7055: --- > Nicholas, I apologize if these findbugs issues inconvenienced you. ... The findbugs issues are minor but the way that the patches got committed ... > ... I would appreciate a review on HDFS-7227. I guess I may not be the best person to review it. Anyway, I just have posted some comments. Please find your favorite reviewers to look at it further. > Thanks also to Yongjun for fixing HDFS-7194 (introduced by me) and HDFS-7169 > (introduced by Nicholas). Yongjun did not work or even comment on HDFS-7169. Typo? > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14169768#comment-14169768 ] Colin Patrick McCabe commented on HDFS-7055: Nicholas, I apologize if these findbugs issues inconvenienced you. I have filed HADOOP-11197 to make test-patch.sh more robust to issues like HADOOP-11178. I would appreciate a review on HDFS-7227. Thanks also to Yongjun for fixing HDFS-7194 (introduced by me) and HDFS-7169 (introduced by Nicholas). > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167484#comment-14167484 ] Tsz Wo Nicholas Sze commented on HDFS-7055: --- bq. You guys know how this works. If Jenkins is green (or the red things in Jenkins can be explained), then it is OK to commit. ... There were indeed -1 javac and -1 findbugs from Jenkins for the patch committed here. However, the explanations given were quite sloppy. - You said "The findbugs warnings are not related" but it turned out that one of the findbugs warnings was related. - You said "Meanwhile diffJavacWarnings.txt is missing, so I can't evaluate where there is an additional warning or not." But the diffJavacWarnings.txt was there for the later builds. Even if the file was missing, we could manually generate it. bq. In this case, it seems that when I left a single slash out of the findbugs file caused us to silently drop some findbugs reports for a while. That seems like a test-patch.sh bug to me. The "friendly" thing to do would be to fix test-patch.sh to flag this condition so that we didn't have to worry about this happening in the future. There are many potential bugs that test-patch.sh is not able to catch. Improving test-patch.sh usually won't get into the priority. bq. In the past I made a bunch of improvements to Jenkins. ... Sure, I believe you did a lot of good work in the past. Thank you! I simply hope that you could make sure the patches are all good before committing them. Is it too much? > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167449#comment-14167449 ] Colin Patrick McCabe commented on HDFS-7055: You guys know how this works. If Jenkins is green (or the red things in Jenkins can be explained), then it is OK to commit. If it later turns out that Jenkins was not as thorough as it could be, or had a bug, then that's a separate issue. In this case, it seems that when I left a single slash out of the findbugs file caused us to silently drop some findbugs reports for a while. That seems like a test-patch.sh bug to me. The "friendly" thing to do would be to fix test-patch.sh to flag this condition so that we didn't have to worry about this happening in the future. In the past I made a bunch of improvements to Jenkins. For example, in HADOOP-10325 I changed it to using a diff-based system for javadoc warnings. I reviewed HADOOP-9186 (which Binglin posted) which gave us warnings when trunk compilation was broken. It sounds like we need another fix to make sure that we are actually getting the findbugs reports we need. > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167316#comment-14167316
]
Tsz Wo Nicholas Sze commented on HDFS-7055:
---
{quote}
> Colin Patrick McCabe, I think you need to be more serious on verifying the
> warnings before committing patches.
That warning did not appear in the newPatchFindbugsWarnings file:
https://builds.apache.org/job/PreCommit-HDFS-Build/8284//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
{quote}
It won't be on the hdfs findbugs output file since this findbugs warning is in
common.
Why there was no Common findbugs output file? It was because the patch
committed by HDFS-6956 (also contributed and committed by [~cmccabe]) broke
findbugsExcludeFile.xml in common.
Colin, I just want to give you a friendly reminder that we all should make sure
the patches are all good (no javac/findbugs warnings, didn't break the build,
etc.) before committing them. Hope that you won't mind.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0
>
> Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch,
> HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png,
> screenshot-get-1mb.png
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167269#comment-14167269 ] Colin Patrick McCabe commented on HDFS-7055: bq. Colin Patrick McCabe, I think you need to be more serious on verifying the warnings before committing patches. That warning did not appear in the newPatchFindbugsWarnings file: https://builds.apache.org/job/PreCommit-HDFS-Build/8284//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Hopefully as the build stabilizes, we will get fewer findbugs false positives and false negatives. I have filed HDFS-7227 to fix the findbugs warning. > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167209#comment-14167209 ] Tsz Wo Nicholas Sze commented on HDFS-7055: --- One more findbugs warning from the patch: - Dodgy code Warnings NP Dereference of the result of readLine() without nullcheck in org.apache.hadoop.tracing.SpanReceiverHost.getUniqueLocalTraceFileName() Bug type NP_DEREFERENCE_OF_READLINE_VALUE (click for details) In class org.apache.hadoop.tracing.SpanReceiverHost In method org.apache.hadoop.tracing.SpanReceiverHost.getUniqueLocalTraceFileName() Value loaded from line At SpanReceiverHost.java:\[line 104] [~cmccabe], I think you need to be more serious on verifying the warnings before committing patches. > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160922#comment-14160922 ] Colin Patrick McCabe commented on HDFS-7055: Thanks for the report. I have filed HDFS-7198 to remove or suppress the javac warning. [~yzhangal] is tackling the findbugs warning in HDFS-7194. > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160758#comment-14160758
]
Tsz Wo Nicholas Sze commented on HDFS-7055:
---
> The findbugs warnings are not related (they're for code issues that already
> exist).
One of the findbugs warnings was from the patch as shown in the [Jenkins
report|https://builds.apache.org/job/PreCommit-HDFS-Build/8284/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html].
- Dmorg.apache.hadoop.hdfs.DFSClient.()
invokes inefficient new String(String) constructor
Bug type DM_STRING_CTOR (click for details)
In class org.apache.hadoop.hdfs.DFSClient
In method org.apache.hadoop.hdfs.DFSClient.()
At DFSClient.java:\[line 3174]
{code}
private static final byte[] PATH =
new String("path").getBytes(Charset.forName("UTF-8"));
{code}
> Meanwhile diffJavacWarnings.txt is missing, so I can't evaluate where there
> is an additional warning or not.
Have you checked the later builds? The
[diffJavacWarnings.txt|https://builds.apache.org/job/PreCommit-HDFS-Build/8284/artifact/patchprocess/diffJavacWarnings.txt]
file was available. The javac warnings were indeed from the patch.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0
>
> Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch,
> HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png,
> screenshot-get-1mb.png
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159163#comment-14159163 ] Hudson commented on HDFS-7055: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1916 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1916/]) HDFS-7055. Add tracing to DFSInputStream (cmccabe) (cmccabe: rev 7f6ed7fe365166e8075359f1d0ad035fa876c70f) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/TraceSamplerFactory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159139#comment-14159139 ] Hudson commented on HDFS-7055: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1891 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1891/]) HDFS-7055. Add tracing to DFSInputStream (cmccabe) (cmccabe: rev 7f6ed7fe365166e8075359f1d0ad035fa876c70f) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/TraceSamplerFactory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159052#comment-14159052 ] Masatake Iwasaki commented on HDFS-7055: bq. Can we discuss this in a follow-on? It seems like we can keep refining the trace point locations as we do more and more testing. Sure. I'll consider the way to control trace more. Thanks for this work! > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14159032#comment-14159032 ] Hudson commented on HDFS-7055: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #700 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/700/]) HDFS-7055. Add tracing to DFSInputStream (cmccabe) (cmccabe: rev 7f6ed7fe365166e8075359f1d0ad035fa876c70f) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/TraceSamplerFactory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158501#comment-14158501 ] Hudson commented on HDFS-7055: -- FAILURE: Integrated in Hadoop-trunk-Commit #6188 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6188/]) HDFS-7055. Add tracing to DFSInputStream (cmccabe) (cmccabe: rev 7f6ed7fe365166e8075359f1d0ad035fa876c70f) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/TraceSamplerFactory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/tracing/SpanReceiverHost.java * hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocalLegacy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.7.0 > > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158344#comment-14158344 ] stack commented on HDFS-7055: - +1 on patch. Went over it again. Clean. Agree it good for now (Looks like it 'works' going by [~iwasakims]'s fancy picture). > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158254#comment-14158254
]
Colin Patrick McCabe commented on HDFS-7055:
bq. I agree but still see 280 spans on .005 patch.
If you trace every read (sampler set to Sampler.ALWAYS), then yes, you will see
a lot of spans. If you want a smaller number of spans, using the
{{ProbabilitySampler}} would be a better choice.
Most of the spans shown in that picture are {{DFSInputStream#byteArrayRead}}.
But this is something that needs to be traced when tracing is on. It's one of
the entry points to the HDFS read path. If we don't trace this entry point, we
have no way of associating any trace spans underneath with the HDFS read that
caused them. In the context of HBase, this would also mean we didn't know
where HBase's spans ended and HDFS's began.
Let's get this in and then we can continue tweaking trace points.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch,
> HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png,
> screenshot-get-1mb.png
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157731#comment-14157731
]
Hadoop QA commented on HDFS-7055:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12672746/screenshot-get-1mb.005.png
against trunk revision 2d8e6e2.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8310//console
This message is automatically generated.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch,
> HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png,
> screenshot-get-1mb.png
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157730#comment-14157730 ] Masatake Iwasaki commented on HDFS-7055: bq. I think we can reduce this by not creating trace spans for every read done via a BlockReader-- only the reads which actually result in data being written from the DN. I agree but still see 280 spans on .005 patch. > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.005.png, > screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155439#comment-14155439 ] Colin Patrick McCabe commented on HDFS-7055: The findbugs warnings are not related (they're for code issues that already exist). > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154383#comment-14154383
]
Hadoop QA commented on HDFS-7055:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12672236/HDFS-7055.005.patch
against trunk revision 17d1202.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1266 javac
compiler warnings (more than the trunk's current 1264 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.crypto.random.TestOsSecureRandom
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8284//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8284//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8284//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8284//console
This message is automatically generated.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch,
> HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.png
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154260#comment-14154260
]
Hadoop QA commented on HDFS-7055:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12672236/HDFS-7055.005.patch
against trunk revision 17d1202.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1266 javac
compiler warnings (more than the trunk's current 1264 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8283//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8283//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8283//console
This message is automatically generated.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch,
> HDFS-7055.004.patch, HDFS-7055.005.patch, screenshot-get-1mb.png
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154161#comment-14154161
]
Colin Patrick McCabe commented on HDFS-7055:
bq. So, if no receiver set up, traces go nowhere?
Yeah.
bq. To see anything, you'd have to load a receiver with the just-added trace
command (i.e.HDFS-6956)?
There's two ways to turn on tracing:
* set {{hadoop.htrace.spanreceiver.classes}} (usually to
{{LocalFileSpanReceiver}} or {{ZipkinSpanReceiver}}
* use "hadoop trace \-add" to add tracing to an already-running daemon.
If you're testing, it's probably easier to just set up your Hadoop XML
configuration so that tracing will be running as soon as the daemon starts up.
This is also the only option for tracing a short-running program like the
FSShell. The dynamic changing is really more for production deployments, where
people might not want this on all the time, but don't want to restart daemons
to turn it on.
bq. If so, that seems cleaner. The alternative where files show up in /tmp if
no receiver is set is less so. Drop this facility from this patch?
Well, the files don't show up in /tmp unless {{LocalFileSpanReceiver}} is
configured. By default, there are no trace span receivers configured, and
nothing shows up in {{/tmp}}.
bq. On the patch, are these defined anywhere "NeverSampler"? Should we make it
so they are? Or should they be toString of the enums?
This is another thing I'd like to add to HTrace. I think HTrace should have
some method that creates a sampler based on a string which is the class name.
But right now that doesn't exist, so I had to add a hacked up version of it to
Hadoop (for now). We can easily take that out once HTrace has it, though.
I'd like to get the basic stuff in and then start shifting more things that we
think could be common code into HTrace.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch,
> HDFS-7055.004.patch, screenshot-get-1mb.png
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153744#comment-14153744 ] stack commented on HDFS-7055: - bq. I think SpanReceiverHost#getUniqueLocalTraceFileName is useful but it should belong to htrace. So, if no receiver set up, traces go nowhere? To see anything, you'd have to load a receiver with the just-added trace command (i.e.HDFS-6956)? If so, that seems cleaner. The alternative where files show up in /tmp if no receiver is set is less so. Drop this facility from this patch? On the patch, are these defined anywhere "NeverSampler"? Should we make it so they are? Or should they be toString of the enums? Otherwise, the patch looks good to me caveat the interjection you and Masatake are figuring. Good stuff. > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153706#comment-14153706 ] Colin Patrick McCabe commented on HDFS-7055: bq. I think SpanReceiverHost#getUniqueLocalTraceFileName is useful but it should belong to htrace. Can I port it to htrace later and remove from hadoop on the next bumping of htrace version? Yeah, absolutely. bq. I attached screenshot of spans for reference. It shows trace of getting 1MB of file by FsShell on pseudo distributed cluster with .004 patch. The trace consists of over 500 spans in this case Setting hadoop.trace.sampler=ProbabilitySampler did not reduce the number of spans above because Trace#startSpan always start span without regarding to sampler when there is ongoing trace. Well, I guess it depends on what you mean by "granular." :) I certainly don't want all trace spans to be activated randomly. We need to see the parent/child relationships between the spans. I think the granularity of individual reads is just about right-- less than that, and we start not being able to see the big picture. More than that, and we can't effectively do random sampling. But you are right that we have too many trace spans here. I thought about this a little more, and I don't think we have to create a trace span for each BlockReader operation. We can just create trace spans for the operations that actually perform I/O to the datanode. I think we can reduce this by not creating trace spans for every read done via a BlockReader-- only the reads which actually result in data being written from the DN. Similarly for BlockReaderLocal, we can trace the times we fill up the buffer, but not every call into BlockReaderLocal. > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153070#comment-14153070
]
Masatake Iwasaki commented on HDFS-7055:
Setting hadoop.trace.sampler=ProbabilitySampler did not reduce the number of
spans above because Trace#startSpan always start span without regarding to
sampler when there is ongoing trace.
{code}
public static TraceScope startSpan(String description, Sampler s,
TraceInfo tinfo) {
Span span = null;
if (isTracing() || s.next(tinfo)) {
span = new MilliSpan(description, tinfo.traceId, tinfo.spanId,
...
{code}
I would like guard for this situation.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch,
> HDFS-7055.004.patch, screenshot-get-1mb.png
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153026#comment-14153026 ] Masatake Iwasaki commented on HDFS-7055: bq. It's fairly granular tracing I attached screenshot of spans for reference. It shows trace of getting 1MB of file by FsShell on pseudo distributed cluster with .004 patch. The trace consists of over 500 spans in this case. > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch, screenshot-get-1mb.png > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153021#comment-14153021 ] Masatake Iwasaki commented on HDFS-7055: I think SpanReceiverHost#getUniqueLocalTraceFileName is useful but it should belong to htrace. Can I port it to htrace later and remove from hadoop on the next bumping of htrace version? > Add tracing to DFSInputStream > - > > Key: HDFS-7055 > URL: https://issues.apache.org/jira/browse/HDFS-7055 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: 2.6.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch, > HDFS-7055.004.patch > > > Add tracing to DFSInputStream. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152334#comment-14152334
]
Hadoop QA commented on HDFS-7055:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12671839/HDFS-7055.004.patch
against trunk revision b3d5d26.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1265 javac
compiler warnings (more than the trunk's current 1263 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.crypto.random.TestOsSecureRandom
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
The following test timeouts occurred in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8254//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8254//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8254//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8254//console
This message is automatically generated.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch,
> HDFS-7055.004.patch
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148404#comment-14148404
]
Hadoop QA commented on HDFS-7055:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12671307/HDFS-7055.003.patch
against trunk revision 9f9a222.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1266 javac
compiler warnings (more than the trunk's current 1264 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService
org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
org.apache.hadoop.yarn.server.resourcemanager.TestRM
org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler
org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8210//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8210//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8210//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8210//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8210//console
This message is automatically generated.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch, HDFS-7055.003.patch
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145627#comment-14145627
]
Colin Patrick McCabe commented on HDFS-7055:
bq. Nit: This exception, if it possible to ask trace for the list of options,
should list the possible options (I can see folks typing in sampler with wrong
case or missing a piece... listing possible options will allow them quickly see
what they have done wrong):
yeah, will add
bq. Nit: Should we have a convention naming spans Colin Patrick McCabe? For
example, method name followed by arg types all in camel case?
readByteArrayIntInt
I'm not sure about naming them according to function calls, since function
calls do change over time. Also, functions like "public int read(long
position, byte[] buffer, int offset, int length)" will start to look like
"Vogon poetry" if we make them into "readLongByteArrayIntInt".
"positionalByteArrayRead" seems a lot clearer to me, and emphasizes the fact
that it includes a "seek." Maybe it makes sense to add a class prefix (like
{{DFSInputStream#positionalByteArrayRead}}?) I think we should discuss span
naming in a follow-up JIRA down the line, once we get all this stuff in.
bq. Patch looks great to me. You gotten any spans out of it? I can try it if
you'd like, no problem.
Yeah, I did some testing with the local file sink. My configuration is here:
https://github.com/cmccabe/hadoop-conf/tree/trace
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145389#comment-14145389
]
Hadoop QA commented on HDFS-7055:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670363/HDFS-7055.002.patch
against trunk revision 5338ac4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1266 javac
compiler warnings (more than the trunk's current 1264 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8166//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8166//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8166//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8166//console
This message is automatically generated.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145221#comment-14145221
]
stack commented on HDFS-7055:
-
bq. One thing to keep in mind here is that if you call Trace.startSpan with
Sampler.NEVER, and there is an existing thread trace span, a subspan will
always be created.
Thanks for mentioning this up front... first thing I stumbled on looking in
code. Its a little confusing but having a comment to explain NEVER in every
span open, it'll get annoying fast.
Nit: This exception, if it possible to ask trace for the list of options,
should list the possible options (I can see folks typing in sampler with wrong
case or missing a piece... listing possible options will allow them quickly see
what they have done wrong):
+ throw new RuntimeException("Can't create sampler " + samplerStr);
Nit: Should we have a convention naming spans [~cmccabe]? For example, method
name followed by arg types all in camel case?
+dfsClient.getTraceScope("byteBufferRead", src);
... would become readByteBuffer and +
dfsClient.getTraceScope("byteArrayRead", src); would be readByteArrayIntInt?
Patch looks great to me. You gotten any spans out of it? I can try it if you'd
like, no problem.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14145113#comment-14145113
]
Colin Patrick McCabe commented on HDFS-7055:
Jenkins says that there is a new findbugs warning, but looking at:
https://builds.apache.org/job/PreCommit-HDFS-Build/8150//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
It says there are 0?
Meanwhile {{diffJavacWarnings.txt}} is missing, so I can't evaluate where there
is an additional warning or not.
Jenkins has been frustrating lately. I will re-trigger this build.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144249#comment-14144249
]
Hadoop QA commented on HDFS-7055:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670363/HDFS-7055.002.patch
against trunk revision 43efdd3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1266 javac
compiler warnings (more than the trunk's current 1264 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.balancer.TestBalancer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8150//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8150//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8150//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8150//console
This message is automatically generated.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142993#comment-14142993
]
Hadoop QA commented on HDFS-7055:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670363/HDFS-7055.002.patch
against trunk revision eab15af.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1266 javac
compiler warnings (more than the trunk's current 1264 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8139//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8139//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8139//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8139//console
This message is automatically generated.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7055) Add tracing to DFSInputStream
[
https://issues.apache.org/jira/browse/HDFS-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142925#comment-14142925
]
Colin Patrick McCabe commented on HDFS-7055:
This patch adds tracing to DFSInputStream. It's fairly granulary tracing, done
at the level of read operations. I also added trace spans to the BlockReaders
(both remote and local) so that the amount of time they spend can be
disentangled from other things. I added tracing to {{BlockSender}} so that the
amount of time we spend sending the block can be seen.
By setting {{dfs.client.trace.sampler}}, users can make their HDFS client
create new trace spans on some requests. The available options are
{{NeverSampler}}, {{AlwaysSampler}}, and {{ProbabilitySampler}}.
One thing to keep in mind here is that if you call {{Trace.startSpan}} with
{{Sampler.NEVER}}, and there is an existing thread trace span, a subspan will
always be created. But if there is no existing thread trace span, none will be
created. So there is no need for if statements and null/not null logic here.
You can always get "pass through" behavior by using {{Sampler.NEVER}}.
> Add tracing to DFSInputStream
> -
>
> Key: HDFS-7055
> URL: https://issues.apache.org/jira/browse/HDFS-7055
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.6.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7055.002.patch
>
>
> Add tracing to DFSInputStream.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
