[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280412#comment-14280412
]
Hudson commented on HDFS-7189:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2026 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2026/])
HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin P. McCabe
via yliu) (yliu: rev c4ccbe62c0857261b197a68c1e03a02e92f21a38)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/EncryptionZoneIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockStorageLocationUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolIterator.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveIterator.java
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0
>
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch,
> HDFS-7189.007.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280351#comment-14280351
]
Hudson commented on HDFS-7189:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #76 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/76/])
HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin P. McCabe
via yliu) (yliu: rev c4ccbe62c0857261b197a68c1e03a02e92f21a38)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/EncryptionZoneIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockStorageLocationUtil.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0
>
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch,
> HDFS-7189.007.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280305#comment-14280305
]
Hudson commented on HDFS-7189:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #72 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/72/])
HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin P. McCabe
via yliu) (yliu: rev c4ccbe62c0857261b197a68c1e03a02e92f21a38)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockStorageLocationUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/EncryptionZoneIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0
>
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch,
> HDFS-7189.007.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280284#comment-14280284
]
Hudson commented on HDFS-7189:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2007 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2007/])
HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin P. McCabe
via yliu) (yliu: rev c4ccbe62c0857261b197a68c1e03a02e92f21a38)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockStorageLocationUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/EncryptionZoneIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveIterator.java
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0
>
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch,
> HDFS-7189.007.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280095#comment-14280095
]
Hudson commented on HDFS-7189:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #809 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/809/])
HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin P. McCabe
via yliu) (yliu: rev c4ccbe62c0857261b197a68c1e03a02e92f21a38)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/EncryptionZoneIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockStorageLocationUtil.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveIterator.java
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0
>
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch,
> HDFS-7189.007.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280081#comment-14280081
]
Hudson commented on HDFS-7189:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #75 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/75/])
HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin P. McCabe
via yliu) (yliu: rev c4ccbe62c0857261b197a68c1e03a02e92f21a38)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockStorageLocationUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/EncryptionZoneIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveIterator.java
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0
>
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch,
> HDFS-7189.007.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279967#comment-14279967
]
Colin Patrick McCabe commented on HDFS-7189:
Thanks, Yi!
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0
>
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch,
> HDFS-7189.007.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279650#comment-14279650
]
Hudson commented on HDFS-7189:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6874 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6874/])
HDFS-7189. Add trace spans for DFSClient metadata operations. (Colin P. McCabe
via yliu) (yliu: rev c4ccbe62c0857261b197a68c1e03a02e92f21a38)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/EncryptionZoneIterator.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockStorageLocationUtil.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CacheDirectiveIterator.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/CachePoolIterator.java
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.7.0
>
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch,
> HDFS-7189.007.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279608#comment-14279608
]
Yi Liu commented on HDFS-7189:
--
+1, Thanks for the contribution.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch,
> HDFS-7189.007.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14279238#comment-14279238
]
Colin Patrick McCabe commented on HDFS-7189:
bq. But I think the spans \[for queryDatanodeForHdfsBlocksMetadata comments\]
should be in the same trace tree and as a child span instead of start a new
trace. So we should pass the trace info (trace id parent span id) to continue
span.
Good idea. That way, if a {{DFSClient#getBlockStorageLocations}} call takes a
long time, the trace might be helpful to show which datanode is lagging.
bq. Some DFSClient operations don't have trace spans, should we add them?
The operations that I didn't add tracing for broke down into two categories:
* Things that we're going to do later in the "add tracing to DFSOutputStream"
change (this is why callAppend isn't traced, append, connectToDN, etc.)
* Things that look like they're not very time-consuming. Like
getCryptoProtocolVersion which just fetches the version out of a
FileEncryptionInfo object.
Actually, though, I do see one or two things where I should add tracing, like
getStoragePolicies. Let me add those.
We can always add tracing to stuff later, so I don't think this should be a
blocker for this going in. And we will certainly adjust the trace points later
based on what we learn... to put more in some places and fewer in others. But
thanks for getting me thinking about this.
bq. Typo? It should be "dst"
Argh. Thanks
bq. In getSrcDstTraceScope, we'd better check dst != null too.
yeah
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278425#comment-14278425
]
Yi Liu commented on HDFS-7189:
--
Thanks Colin for the patch. Overall looks very good, only following comments:
*1).* Besides {{DFSClient}}, I see the patch passes the {{traceSampler}} to
some other classes, for example _BlockStorageLocationUtil.java_. The reason is
{{queryDatanodesForHdfsBlocksMetadata}} is invoked by
{{DFSClient#getBlockStorageLocations}}, but it uses a thread pool to call
{{VolumeBlockLocationCallable}}, so it's in different thread context. If we
don't pass {{traceSampler}}, then we will get {{NullScope}} for {{startSpan}}.
But I think the span should be in the same trace tree and as a child span
instead of start a new trace. So I think we should pass the trace info (trace
id parent span id) to continue span.
For {{CacheDirectiveIterator}}, {{CachePoolIterator}} and
{{EncryptionZoneIterato}}, it indeed can be used by user in new places of
different threads, so it's OK we pass {{traceSampler}} and can start new trace
if it's indeed in different thread.
*2).*
Some DFSClient operations don't have trace spans, should we add them?
*3).*
{code}
private static final byte[] DST = "src".getBytes(Charset.forName("UTF-8"));
{code}
Typo? It should be {{"dst"}}
*4).*
In {{getSrcDstTraceScope}}, we should also check {{dst != null}}
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278264#comment-14278264
]
Hadoop QA commented on HDFS-7189:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692406/HDFS-7189.006.patch
against trunk revision 6464a89.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1210 javac
compiler warnings (more than the trunk's current 1206 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestFileTruncate
org.apache.hadoop.hdfs.qjournal.TestSecureNNWithQJM
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9219//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9219//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9219//console
This message is automatically generated.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch, HDFS-7189.006.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277826#comment-14277826
]
Hadoop QA commented on HDFS-7189:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692303/HDFS-7189.005.patch
against trunk revision 446545c.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1220 javac
compiler warnings (more than the trunk's current 1206 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9211//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/9211//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9211//console
This message is automatically generated.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch, HDFS-7189.005.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276445#comment-14276445
]
Hadoop QA commented on HDFS-7189:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12692143/HDFS-7189.004.patch
against trunk revision f92e503.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9206//console
This message is automatically generated.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276440#comment-14276440
]
Colin Patrick McCabe commented on HDFS-7189:
rebased.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch,
> HDFS-7189.004.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14272102#comment-14272102
]
Colin Patrick McCabe commented on HDFS-7189:
Hi Yi, I'm going to repost in a bit. Thanks for your patience!
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270493#comment-14270493
]
Hadoop QA commented on HDFS-7189:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673205/HDFS-7189.003.patch
against trunk revision ae91b13.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9163//console
This message is automatically generated.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270490#comment-14270490
]
Yi Liu commented on HDFS-7189:
--
Colin, could you rebase the patch? Then we can continue the review.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164677#comment-14164677
]
Hadoop QA commented on HDFS-7189:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673205/HDFS-7189.003.patch
against trunk revision df3becf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1338 javac
compiler warnings (more than the trunk's current 1266 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
org.apache.hadoop.hdfs.TestRollingUpgradeRollback
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8363//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8363//artifact/patchprocess/patchReleaseAuditProblems.txt
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8363//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8363//console
This message is automatically generated.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161145#comment-14161145
]
Hadoop QA commented on HDFS-7189:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673205/HDFS-7189.003.patch
against trunk revision 8dc6abf.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8337//console
This message is automatically generated.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161137#comment-14161137
]
Colin Patrick McCabe commented on HDFS-7189:
Build failed due to BUILDS-26, retriggering
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14161039#comment-14161039
]
Hadoop QA commented on HDFS-7189:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12673205/HDFS-7189.003.patch
against trunk revision 8099de2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8335//console
This message is automatically generated.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch, HDFS-7189.003.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160998#comment-14160998
]
Colin Patrick McCabe commented on HDFS-7189:
bq. \[is adding unwrapRemoteException to getStoragePolicies\] a bug fix?
Hmm. Good question. I looked into this a little more, and I think I will skip
adding new invocations of {{unwrapRemoteException}} in this patch. The
unwrapping is only needed when the NameNode actually throws one of those
exceptions, but I don't think that can happen for {{getStoragePolicies}} or
many of the other functions here. Plus, adding that stuff muddies the
waters... it would be better to do it in a separate patch than to combine it
with this one.
bq. Removing checkOpen(); in delete is intentional ?
Ah, but the one-argument version of {{delete}} now calls another override of
the function, which then calls {{checkOpen}}. So it should be OK.
bq. Is this intentional... calling trace getCurrentEditLogTxid though its in
getInotifyEventStream ... I suppose it is given it actually does do
getCurrentEditLogTxid
I think we should, since we want to know about this source of activity. We
want to know what the performance impact of inotify is.
I also fixed a findbugs warning. Reposting
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158940#comment-14158940
]
stack commented on HDFS-7189:
-
This a bug fix?
1860TraceScope scope = Trace.startSpan("getStoragePolicies",
traceSampler);
1861try {
1862 return namenode.getStoragePolicies();
1863} catch (RemoteException e) {
1864 throw e.unwrapRemoteException(AccessControlException.class,
1865 FileNotFoundException.class,
1866 SafeModeException.class,
1867 NSQuotaExceededException.class,
1868 UnresolvedPathException.class,
1869 SnapshotAccessControlException.class);
1870} finally {
.. and in listCorruptFileBlocks etc. ?
Removing checkOpen(); in delete is intentional ?
Is this intentional... calling trace getCurrentEditLogTxid though its in
getInotifyEventStream ... I suppose it is given it actually does do
getCurrentEditLogTxid
public DFSInotifyEventInputStream getInotifyEventStream() throws IOException {
3350long txid;
3351TraceScope scope = Trace.startSpan("getCurrentEditLogTxid",
traceSampler);
3352try {
3353 txid = namenode.getCurrentEditLogTxid();
Adding src/dst annotations is nice touch
Patch is great.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7189) Add trace spans for DFSClient metadata operations
[
https://issues.apache.org/jira/browse/HDFS-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158685#comment-14158685
]
Hadoop QA commented on HDFS-7189:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12672866/HDFS-7189.001.patch
against trunk revision 7f6ed7f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8319//console
This message is automatically generated.
> Add trace spans for DFSClient metadata operations
> -
>
> Key: HDFS-7189
> URL: https://issues.apache.org/jira/browse/HDFS-7189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: 2.7.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-7189.001.patch
>
>
> We should add trace spans for DFSClient metadata operations. For example,
> {{DFSClient#rename}} should have a trace span, etc. etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
