[jira] [Commented] (HDFS-5956) A file size is multiplied by the replication factor in 'hdfs oiv -p FileDistribution' option
[
https://issues.apache.org/jira/browse/HDFS-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902341#comment-13902341
]
Akira AJISAKA commented on HDFS-5956:
-
The test failure is not related to the patch.
> A file size is multiplied by the replication factor in 'hdfs oiv -p
> FileDistribution' option
>
>
> Key: HDFS-5956
> URL: https://issues.apache.org/jira/browse/HDFS-5956
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 3.0.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: HDFS-5956.patch
>
>
> In FileDistributionCalculator.java,
> {code}
> long fileSize = 0;
> for (BlockProto b : f.getBlocksList()) {
> fileSize += b.getNumBytes() * f.getReplication();
> }
> maxFileSize = Math.max(fileSize, maxFileSize);
> totalSpace += fileSize;
> {code}
> should be
> {code}
> long fileSize = 0;
> for (BlockProto b : f.getBlocksList()) {
> fileSize += b.getNumBytes();
> }
> maxFileSize = Math.max(fileSize, maxFileSize);
> totalSpace += fileSize * f.getReplication();
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5956) A file size is multiplied by the replication factor in 'hdfs oiv -p FileDistribution' option
[
https://issues.apache.org/jira/browse/HDFS-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902328#comment-13902328
]
Hadoop QA commented on HDFS-5956:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629167/HDFS-5956.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6163//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6163//console
This message is automatically generated.
> A file size is multiplied by the replication factor in 'hdfs oiv -p
> FileDistribution' option
>
>
> Key: HDFS-5956
> URL: https://issues.apache.org/jira/browse/HDFS-5956
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 3.0.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: HDFS-5956.patch
>
>
> In FileDistributionCalculator.java,
> {code}
> long fileSize = 0;
> for (BlockProto b : f.getBlocksList()) {
> fileSize += b.getNumBytes() * f.getReplication();
> }
> maxFileSize = Math.max(fileSize, maxFileSize);
> totalSpace += fileSize;
> {code}
> should be
> {code}
> long fileSize = 0;
> for (BlockProto b : f.getBlocksList()) {
> fileSize += b.getNumBytes();
> }
> maxFileSize = Math.max(fileSize, maxFileSize);
> totalSpace += fileSize * f.getReplication();
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902288#comment-13902288
]
Hadoop QA commented on HDFS-5776:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629137/HDFS-5776v18.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6160//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6160//console
This message is automatically generated.
> Support 'hedged' reads in DFSClient
> ---
>
> Key: HDFS-5776
> URL: https://issues.apache.org/jira/browse/HDFS-5776
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: HDFS-5776-v10.txt, HDFS-5776-v11.txt, HDFS-5776-v12.txt,
> HDFS-5776-v12.txt, HDFS-5776-v13.wip.txt, HDFS-5776-v14.txt,
> HDFS-5776-v15.txt, HDFS-5776-v17.txt, HDFS-5776-v17.txt, HDFS-5776-v2.txt,
> HDFS-5776-v3.txt, HDFS-5776-v4.txt, HDFS-5776-v5.txt, HDFS-5776-v6.txt,
> HDFS-5776-v7.txt, HDFS-5776-v8.txt, HDFS-5776-v9.txt, HDFS-5776.txt,
> HDFS-5776v18.txt
>
>
> This is a placeholder of hdfs related stuff backport from
> https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" &
> "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read
> ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we
> could export the interested metric valus into client system(e.g. HBase's
> regionserver metric).
> The core logic is in pread code path, we decide to goto the original
> fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per
> the above config items.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5866) '-maxSize' and '-step' option fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-5866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902287#comment-13902287
]
Hadoop QA commented on HDFS-5866:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629161/HDFS-5866.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6162//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6162//console
This message is automatically generated.
> '-maxSize' and '-step' option fail in OfflineImageViewer
>
>
> Key: HDFS-5866
> URL: https://issues.apache.org/jira/browse/HDFS-5866
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-5866.2.patch, HDFS-5866.patch
>
>
> Executing -step or/and -maxSize option will get the following error:
> {code}
> $ hdfs oiv -p FileDistribution -step 102400 -i input -o output
> Error parsing command-line options:
> Usage: bin/hdfs oiv [OPTIONS] -i INPUTFILE -o OUTPUTFILE
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5780) TestRBWBlockInvalidation times out intemittently on branch-2
[
https://issues.apache.org/jira/browse/HDFS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902275#comment-13902275
]
Hadoop QA commented on HDFS-5780:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629132/HDFS-5780.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6159//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6159//console
This message is automatically generated.
> TestRBWBlockInvalidation times out intemittently on branch-2
>
>
> Key: HDFS-5780
> URL: https://issues.apache.org/jira/browse/HDFS-5780
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: HDFS-5780.patch, HDFS-5780.patch
>
>
> i recently found out that the test
> TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN times
> out intermittently.
> I am using Fedora, JDK7
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5698) Use protobuf to serialize / deserialize FSImage
[ https://issues.apache.org/jira/browse/HDFS-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902264#comment-13902264 ] Hudson commented on HDFS-5698: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5175 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5175/]) HDFS-5698. Update CHANGES.txt to move breakdown of tasks to correct release (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568582) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Use protobuf to serialize / deserialize FSImage > --- > > Key: HDFS-5698 > URL: https://issues.apache.org/jira/browse/HDFS-5698 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5698-branch2.000.patch, HDFS-5698-design.pdf, > HDFS-5698.000.patch, HDFS-5698.001.patch, HDFS-5698.002.patch, > HDFS-5698.003.patch, HDFS-5698.004.patch, HDFS-5698.005.patch, > HDFS-5698.006.patch, HDFS-5698.007.patch > > > Currently, the code serializes FSImage using in-house serialization > mechanisms. There are a couple disadvantages of the current approach: > # Mixing the responsibility of reconstruction and serialization / > deserialization. The current code paths of serialization / deserialization > have spent a lot of effort on maintaining compatibility. What is worse is > that they are mixed with the complex logic of reconstructing the namespace, > making the code difficult to follow. > # Poor documentation of the current FSImage format. The format of the FSImage > is practically defined by the implementation. An bug in implementation means > a bug in the specification. Furthermore, it also makes writing third-party > tools quite difficult. > # Changing schemas is non-trivial. Adding a field in FSImage requires bumping > the layout version every time. Bumping out layout version requires (1) the > users to explicitly upgrade the clusters, and (2) putting new code to > maintain backward compatibility. > This jira proposes to use protobuf to serialize the FSImage. Protobuf has > been used to serialize / deserialize the RPC message in Hadoop. > Protobuf addresses all the above problems. It clearly separates the > responsibility of serialization and reconstructing the namespace. The > protobuf files document the current format of the FSImage. The developers now > can add optional fields with ease, since the old code can always read the new > FSImage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5953) TestBlockReaderFactory fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902262#comment-13902262
]
Hudson commented on HDFS-5953:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5175 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5175/])
HDFS-5953. TestBlockReaderFactory fails in trunk. (Akira Ajisaka via wang)
(wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568568)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderFactory.java
> TestBlockReaderFactory fails in trunk
> -
>
> Key: HDFS-5953
> URL: https://issues.apache.org/jira/browse/HDFS-5953
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Akira AJISAKA
> Fix For: 2.4.0
>
> Attachments: HDFS-5953.patch
>
>
> From
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1673/testReport/junit/org.apache.hadoop.hdfs/TestBlockReaderFactory/testFallbackFromShortCircuitToUnixDomainTraffic/
> :
> {code}
> java.lang.RuntimeException: Although a UNIX domain socket path is configured
> as
> /tmp/socks.1392383436573.1418778351/testFallbackFromShortCircuitToUnixDomainTraffic._PORT,
> we cannot start a localDataXceiverServer because libhadoop cannot be loaded.
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:601)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:573)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:769)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:315)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1864)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1764)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1243)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:699)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:359)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340)
> at
> org.apache.hadoop.hdfs.TestBlockReaderFactory.testFallbackFromShortCircuitToUnixDomainTraffic(TestBlockReaderFactory.java:99)
> {code}
> This test failure can be reproduced locally (on Mac).
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5749) Access time of HDFS directories stays at 1969-12-31
[ https://issues.apache.org/jira/browse/HDFS-5749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902263#comment-13902263 ] Hudson commented on HDFS-5749: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5175 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5175/]) HDFS-5749. Web UI does not show up during the period of loading FSImage. (Contributed by Haohui Mai) (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568578) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.js > Access time of HDFS directories stays at 1969-12-31 > --- > > Key: HDFS-5749 > URL: https://issues.apache.org/jira/browse/HDFS-5749 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > > Modify FsShell so that "fs -lsr" can show access time in addition to > modification time, the access time stays at 1969-12-31. This means the access > time is not set up initially. Filing this jira to fix this issue. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5716) Allow WebHDFS to use pluggable authentication filter
[ https://issues.apache.org/jira/browse/HDFS-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902265#comment-13902265 ] Hudson commented on HDFS-5716: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5175 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5175/]) HDFS-5716. Allow WebHDFS to use pluggable authentication filter. Contributed by Haohui Mai (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568547) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsWithAuthenticationFilter.java > Allow WebHDFS to use pluggable authentication filter > > > Key: HDFS-5716 > URL: https://issues.apache.org/jira/browse/HDFS-5716 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.4.0 > > Attachments: HDFS-5716.000.patch, HDFS-5716.001.patch, > HDFS-5716.002.patch > > > Currently the WebHDFS server only uses SPNEGO filters for authentication > purposes. The design serves the purpose well when SPNEGO is available, > however, other services (e.g., the web UI) where SPNEGO might be unavailable > can also use WebHDFS. > Note that the web UI and the WebHDFS are served by the same Jetty server on > the same port. The Web UI can use customized authentication filters, it might > make sense to allow WebHDFS to take customized authentication filters as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5952) Create a tool to run data analysis on the PB format fsimage
[ https://issues.apache.org/jira/browse/HDFS-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902248#comment-13902248 ] Akira AJISAKA commented on HDFS-5952: - Rethinking this idea, it is good for data analysis, but not for troubleshooting. It needs too much cost to run Hive/Pig jobs when an cluster is in trouble. Therefore, a tool to dump fsimage into text format is still needed. The tool will output two text files: * files/dirs information * snapshot diffs and users can analyze namespaces or lsr to snapshots by tools such as SQLite. > Create a tool to run data analysis on the PB format fsimage > --- > > Key: HDFS-5952 > URL: https://issues.apache.org/jira/browse/HDFS-5952 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 3.0.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > > Delimited processor in OfflineImageViewer is not supported after HDFS-5698 > was merged. > The motivation of delimited processor is to run data analysis on the fsimage, > therefore, there might be more values to create a tool for Hive or Pig that > reads the PB format fsimage directly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5759) Web UI does not show up during the period of loading FSImage
[ https://issues.apache.org/jira/browse/HDFS-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5759: Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Target Version/s: 2.4.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed this to trunk and branch-2. Thanks for the contribution [~wheat9]. > Web UI does not show up during the period of loading FSImage > > > Key: HDFS-5759 > URL: https://issues.apache.org/jira/browse/HDFS-5759 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5759.000.patch, HDFS-5759.001.patch, screenshot.png > > > The JavaScript code retrieves several JMX responses to construct the UI. Some > JMX responses (e.g., FSNamesystemState) are unavailable during the period of > loading FSImage, leaving the UI as a blank page to the users. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902237#comment-13902237 ] Masatake Iwasaki commented on HDFS-5274: Thanks [~stack]! I'm on it. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, > Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5759) Web UI does not show up during the period of loading FSImage
[ https://issues.apache.org/jira/browse/HDFS-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902230#comment-13902230 ] Arpit Agarwal commented on HDFS-5759: - +1 for the patch, will commit this shortly. Thanks Haohui. > Web UI does not show up during the period of loading FSImage > > > Key: HDFS-5759 > URL: https://issues.apache.org/jira/browse/HDFS-5759 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5759.000.patch, HDFS-5759.001.patch, screenshot.png > > > The JavaScript code retrieves several JMX responses to construct the UI. Some > JMX responses (e.g., FSNamesystemState) are unavailable during the period of > loading FSImage, leaving the UI as a blank page to the users. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5759) Web UI does not show up during the period of loading FSImage
[ https://issues.apache.org/jira/browse/HDFS-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5759: - Attachment: HDFS-5759.001.patch Reword "Redirect to the page of startup progress" to "Redirecting to the Startup Progress page" based on [~arpitagarwal]'s comments. > Web UI does not show up during the period of loading FSImage > > > Key: HDFS-5759 > URL: https://issues.apache.org/jira/browse/HDFS-5759 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5759.000.patch, HDFS-5759.001.patch, screenshot.png > > > The JavaScript code retrieves several JMX responses to construct the UI. Some > JMX responses (e.g., FSNamesystemState) are unavailable during the period of > loading FSImage, leaving the UI as a blank page to the users. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5953) TestBlockReaderFactory fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-5953:
--
Resolution: Fixed
Fix Version/s: 2.4.0
Status: Resolved (was: Patch Available)
Committed to trunk and branch-2. Thanks for the patch Akira and report Ted.
> TestBlockReaderFactory fails in trunk
> -
>
> Key: HDFS-5953
> URL: https://issues.apache.org/jira/browse/HDFS-5953
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Akira AJISAKA
> Fix For: 2.4.0
>
> Attachments: HDFS-5953.patch
>
>
> From
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1673/testReport/junit/org.apache.hadoop.hdfs/TestBlockReaderFactory/testFallbackFromShortCircuitToUnixDomainTraffic/
> :
> {code}
> java.lang.RuntimeException: Although a UNIX domain socket path is configured
> as
> /tmp/socks.1392383436573.1418778351/testFallbackFromShortCircuitToUnixDomainTraffic._PORT,
> we cannot start a localDataXceiverServer because libhadoop cannot be loaded.
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:601)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:573)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:769)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:315)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1864)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1764)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1243)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:699)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:359)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340)
> at
> org.apache.hadoop.hdfs.TestBlockReaderFactory.testFallbackFromShortCircuitToUnixDomainTraffic(TestBlockReaderFactory.java:99)
> {code}
> This test failure can be reproduced locally (on Mac).
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5953) TestBlockReaderFactory fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902213#comment-13902213
]
Andrew Wang commented on HDFS-5953:
---
+1 will commit. I also changed the hdfs trunk build to also build common native
libraries, since we really should be running these tests rather than skipping
them.
> TestBlockReaderFactory fails in trunk
> -
>
> Key: HDFS-5953
> URL: https://issues.apache.org/jira/browse/HDFS-5953
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Akira AJISAKA
> Attachments: HDFS-5953.patch
>
>
> From
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1673/testReport/junit/org.apache.hadoop.hdfs/TestBlockReaderFactory/testFallbackFromShortCircuitToUnixDomainTraffic/
> :
> {code}
> java.lang.RuntimeException: Although a UNIX domain socket path is configured
> as
> /tmp/socks.1392383436573.1418778351/testFallbackFromShortCircuitToUnixDomainTraffic._PORT,
> we cannot start a localDataXceiverServer because libhadoop cannot be loaded.
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:601)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:573)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:769)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:315)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1864)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1764)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1243)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:699)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:359)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340)
> at
> org.apache.hadoop.hdfs.TestBlockReaderFactory.testFallbackFromShortCircuitToUnixDomainTraffic(TestBlockReaderFactory.java:99)
> {code}
> This test failure can be reproduced locally (on Mac).
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5759) Web UI does not show up during the period of loading FSImage
[ https://issues.apache.org/jira/browse/HDFS-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902204#comment-13902204 ] Jing Zhao commented on HDFS-5759: - +1 > Web UI does not show up during the period of loading FSImage > > > Key: HDFS-5759 > URL: https://issues.apache.org/jira/browse/HDFS-5759 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5759.000.patch, screenshot.png > > > The JavaScript code retrieves several JMX responses to construct the UI. Some > JMX responses (e.g., FSNamesystemState) are unavailable during the period of > loading FSImage, leaving the UI as a blank page to the users. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5847) Consolidate INodeReference into a separate section
[ https://issues.apache.org/jira/browse/HDFS-5847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5847: Target Version/s: 2.4.0 Fix Version/s: 2.4.0 I merged this to branch-2. > Consolidate INodeReference into a separate section > -- > > Key: HDFS-5847 > URL: https://issues.apache.org/jira/browse/HDFS-5847 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Haohui Mai >Assignee: Jing Zhao > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5847.000.patch, HDFS-5847.001.patch, > HDFS-5847.002.patch > > > Currently each INodeDirectorySection.Entry contains variable numbers of > INodeReference entries. The INodeReference entries are inlined, therefore it > is difficult to quickly navigate through a INodeDirectorySection.Entry. > Skipping through a INodeDirectorySection.Entry without parsing is essential > to parse these entries in parallel. > This jira proposes to consolidate INodeReferences into a section and give > each of them an ID. The INodeDirectorySection.Entry can store the list of the > IDs as a repeated field. That way we can leverage the existing code in > protobuf to quickly skip through a INodeDirectorySection.Entry. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5915) Refactor FSImageFormatProtobuf to simplify cross section reads
[
https://issues.apache.org/jira/browse/HDFS-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5915:
Target Version/s: 2.4.0 (was: 3.0.0)
Fix Version/s: 2.4.0
I merged this to branch-2.
> Refactor FSImageFormatProtobuf to simplify cross section reads
> --
>
> Key: HDFS-5915
> URL: https://issues.apache.org/jira/browse/HDFS-5915
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HDFS-5915.000.patch, HDFS-5915.001.patch
>
>
> The PB-based FSImage puts the user name and the group name into a separate
> section for deduplication. This jira refactor the code so that it is easier
> to apply the same techniques for other types of data (e.g.,
> {{INodeReference}})
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5716) Allow WebHDFS to use pluggable authentication filter
[ https://issues.apache.org/jira/browse/HDFS-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5716: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Allow WebHDFS to use pluggable authentication filter > > > Key: HDFS-5716 > URL: https://issues.apache.org/jira/browse/HDFS-5716 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5716.000.patch, HDFS-5716.001.patch, > HDFS-5716.002.patch > > > Currently the WebHDFS server only uses SPNEGO filters for authentication > purposes. The design serves the purpose well when SPNEGO is available, > however, other services (e.g., the web UI) where SPNEGO might be unavailable > can also use WebHDFS. > Note that the web UI and the WebHDFS are served by the same Jetty server on > the same port. The Web UI can use customized authentication filters, it might > make sense to allow WebHDFS to take customized authentication filters as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5716) Allow WebHDFS to use pluggable authentication filter
[ https://issues.apache.org/jira/browse/HDFS-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902188#comment-13902188 ] Brandon Li commented on HDFS-5716: -- The new patch has just comment change and got +1 from Jinkins last time already. The test failures are unrelated. I've committed the patch. Thank you, Haohui. > Allow WebHDFS to use pluggable authentication filter > > > Key: HDFS-5716 > URL: https://issues.apache.org/jira/browse/HDFS-5716 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.4.0 > > Attachments: HDFS-5716.000.patch, HDFS-5716.001.patch, > HDFS-5716.002.patch > > > Currently the WebHDFS server only uses SPNEGO filters for authentication > purposes. The design serves the purpose well when SPNEGO is available, > however, other services (e.g., the web UI) where SPNEGO might be unavailable > can also use WebHDFS. > Note that the web UI and the WebHDFS are served by the same Jetty server on > the same port. The Web UI can use customized authentication filters, it might > make sense to allow WebHDFS to take customized authentication filters as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5716) Allow WebHDFS to use pluggable authentication filter
[ https://issues.apache.org/jira/browse/HDFS-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5716: - Fix Version/s: 2.4.0 > Allow WebHDFS to use pluggable authentication filter > > > Key: HDFS-5716 > URL: https://issues.apache.org/jira/browse/HDFS-5716 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.4.0 > > Attachments: HDFS-5716.000.patch, HDFS-5716.001.patch, > HDFS-5716.002.patch > > > Currently the WebHDFS server only uses SPNEGO filters for authentication > purposes. The design serves the purpose well when SPNEGO is available, > however, other services (e.g., the web UI) where SPNEGO might be unavailable > can also use WebHDFS. > Note that the web UI and the WebHDFS are served by the same Jetty server on > the same port. The Web UI can use customized authentication filters, it might > make sense to allow WebHDFS to take customized authentication filters as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5866) '-maxSize' and '-step' option fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-5866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-5866:
Status: Open (was: Patch Available)
> '-maxSize' and '-step' option fail in OfflineImageViewer
>
>
> Key: HDFS-5866
> URL: https://issues.apache.org/jira/browse/HDFS-5866
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 2.2.0, 3.0.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-5866.2.patch, HDFS-5866.patch
>
>
> Executing -step or/and -maxSize option will get the following error:
> {code}
> $ hdfs oiv -p FileDistribution -step 102400 -i input -o output
> Error parsing command-line options:
> Usage: bin/hdfs oiv [OPTIONS] -i INPUTFILE -o OUTPUTFILE
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5803) TestBalancer.testBalancer0 fails
[
https://issues.apache.org/jira/browse/HDFS-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902184#comment-13902184
]
Chen He commented on HDFS-5803:
---
Hi [~kihwal]
Thank you for the suggestion. Here is my data.
TestBalancer takes:
50.33s on 0.23 build
67.96s on trunk build
Each value is an average of 3 sample runs.
> TestBalancer.testBalancer0 fails
>
>
> Key: HDFS-5803
> URL: https://issues.apache.org/jira/browse/HDFS-5803
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Mit Desai
>Assignee: Chen He
> Attachments: HDFS-5803.patch
>
>
> The test testBalancer0 fails on branch 2. Below is the stack trace
> {noformat}
> java.util.concurrent.TimeoutException: Cluster failed to reached expected
> values of totalSpace (current: 1500, expected: 1500), or usedSpace (current:
> 280, expected: 300), in more than 2 msec.
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForHeartBeat(TestBalancer.java:245)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:375)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:359)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.twoNodeTest(TestBalancer.java:404)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0Internal(TestBalancer.java:448)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0(TestBalancer.java:442)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5956) A file size is multiplied by the replication factor in 'hdfs oiv -p FileDistribution' option
[
https://issues.apache.org/jira/browse/HDFS-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-5956:
Status: Patch Available (was: Open)
> A file size is multiplied by the replication factor in 'hdfs oiv -p
> FileDistribution' option
>
>
> Key: HDFS-5956
> URL: https://issues.apache.org/jira/browse/HDFS-5956
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 3.0.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: HDFS-5956.patch
>
>
> In FileDistributionCalculator.java,
> {code}
> long fileSize = 0;
> for (BlockProto b : f.getBlocksList()) {
> fileSize += b.getNumBytes() * f.getReplication();
> }
> maxFileSize = Math.max(fileSize, maxFileSize);
> totalSpace += fileSize;
> {code}
> should be
> {code}
> long fileSize = 0;
> for (BlockProto b : f.getBlocksList()) {
> fileSize += b.getNumBytes();
> }
> maxFileSize = Math.max(fileSize, maxFileSize);
> totalSpace += fileSize * f.getReplication();
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5956) A file size is multiplied by the replication factor in 'hdfs oiv -p FileDistribution' option
[
https://issues.apache.org/jira/browse/HDFS-5956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-5956:
Attachment: HDFS-5956.patch
Attaching a patch.
> A file size is multiplied by the replication factor in 'hdfs oiv -p
> FileDistribution' option
>
>
> Key: HDFS-5956
> URL: https://issues.apache.org/jira/browse/HDFS-5956
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 3.0.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Labels: newbie
> Attachments: HDFS-5956.patch
>
>
> In FileDistributionCalculator.java,
> {code}
> long fileSize = 0;
> for (BlockProto b : f.getBlocksList()) {
> fileSize += b.getNumBytes() * f.getReplication();
> }
> maxFileSize = Math.max(fileSize, maxFileSize);
> totalSpace += fileSize;
> {code}
> should be
> {code}
> long fileSize = 0;
> for (BlockProto b : f.getBlocksList()) {
> fileSize += b.getNumBytes();
> }
> maxFileSize = Math.max(fileSize, maxFileSize);
> totalSpace += fileSize * f.getReplication();
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5716) Allow WebHDFS to use pluggable authentication filter
[
https://issues.apache.org/jira/browse/HDFS-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902178#comment-13902178
]
Hadoop QA commented on HDFS-5716:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629158/HDFS-5716.002.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6161//console
This message is automatically generated.
> Allow WebHDFS to use pluggable authentication filter
>
>
> Key: HDFS-5716
> URL: https://issues.apache.org/jira/browse/HDFS-5716
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5716.000.patch, HDFS-5716.001.patch,
> HDFS-5716.002.patch
>
>
> Currently the WebHDFS server only uses SPNEGO filters for authentication
> purposes. The design serves the purpose well when SPNEGO is available,
> however, other services (e.g., the web UI) where SPNEGO might be unavailable
> can also use WebHDFS.
> Note that the web UI and the WebHDFS are served by the same Jetty server on
> the same port. The Web UI can use customized authentication filters, it might
> make sense to allow WebHDFS to take customized authentication filters as well.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5698) Use protobuf to serialize / deserialize FSImage
[ https://issues.apache.org/jira/browse/HDFS-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902176#comment-13902176 ] Arpit Agarwal commented on HDFS-5698: - Merge was committed as svn r1568548. > Use protobuf to serialize / deserialize FSImage > --- > > Key: HDFS-5698 > URL: https://issues.apache.org/jira/browse/HDFS-5698 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5698-branch2.000.patch, HDFS-5698-design.pdf, > HDFS-5698.000.patch, HDFS-5698.001.patch, HDFS-5698.002.patch, > HDFS-5698.003.patch, HDFS-5698.004.patch, HDFS-5698.005.patch, > HDFS-5698.006.patch, HDFS-5698.007.patch > > > Currently, the code serializes FSImage using in-house serialization > mechanisms. There are a couple disadvantages of the current approach: > # Mixing the responsibility of reconstruction and serialization / > deserialization. The current code paths of serialization / deserialization > have spent a lot of effort on maintaining compatibility. What is worse is > that they are mixed with the complex logic of reconstructing the namespace, > making the code difficult to follow. > # Poor documentation of the current FSImage format. The format of the FSImage > is practically defined by the implementation. An bug in implementation means > a bug in the specification. Furthermore, it also makes writing third-party > tools quite difficult. > # Changing schemas is non-trivial. Adding a field in FSImage requires bumping > the layout version every time. Bumping out layout version requires (1) the > users to explicitly upgrade the clusters, and (2) putting new code to > maintain backward compatibility. > This jira proposes to use protobuf to serialize the FSImage. Protobuf has > been used to serialize / deserialize the RPC message in Hadoop. > Protobuf addresses all the above problems. It clearly separates the > responsibility of serialization and reconstructing the namespace. The > protobuf files document the current format of the FSImage. The developers now > can add optional fields with ease, since the old code can always read the new > FSImage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5698) Use protobuf to serialize / deserialize FSImage
[ https://issues.apache.org/jira/browse/HDFS-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5698: Target Version/s: 2.4.0 > Use protobuf to serialize / deserialize FSImage > --- > > Key: HDFS-5698 > URL: https://issues.apache.org/jira/browse/HDFS-5698 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5698-branch2.000.patch, HDFS-5698-design.pdf, > HDFS-5698.000.patch, HDFS-5698.001.patch, HDFS-5698.002.patch, > HDFS-5698.003.patch, HDFS-5698.004.patch, HDFS-5698.005.patch, > HDFS-5698.006.patch, HDFS-5698.007.patch > > > Currently, the code serializes FSImage using in-house serialization > mechanisms. There are a couple disadvantages of the current approach: > # Mixing the responsibility of reconstruction and serialization / > deserialization. The current code paths of serialization / deserialization > have spent a lot of effort on maintaining compatibility. What is worse is > that they are mixed with the complex logic of reconstructing the namespace, > making the code difficult to follow. > # Poor documentation of the current FSImage format. The format of the FSImage > is practically defined by the implementation. An bug in implementation means > a bug in the specification. Furthermore, it also makes writing third-party > tools quite difficult. > # Changing schemas is non-trivial. Adding a field in FSImage requires bumping > the layout version every time. Bumping out layout version requires (1) the > users to explicitly upgrade the clusters, and (2) putting new code to > maintain backward compatibility. > This jira proposes to use protobuf to serialize the FSImage. Protobuf has > been used to serialize / deserialize the RPC message in Hadoop. > Protobuf addresses all the above problems. It clearly separates the > responsibility of serialization and reconstructing the namespace. The > protobuf files document the current format of the FSImage. The developers now > can add optional fields with ease, since the old code can always read the new > FSImage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5698) Use protobuf to serialize / deserialize FSImage
[ https://issues.apache.org/jira/browse/HDFS-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5698: Fix Version/s: 2.4.0 > Use protobuf to serialize / deserialize FSImage > --- > > Key: HDFS-5698 > URL: https://issues.apache.org/jira/browse/HDFS-5698 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5698-branch2.000.patch, HDFS-5698-design.pdf, > HDFS-5698.000.patch, HDFS-5698.001.patch, HDFS-5698.002.patch, > HDFS-5698.003.patch, HDFS-5698.004.patch, HDFS-5698.005.patch, > HDFS-5698.006.patch, HDFS-5698.007.patch > > > Currently, the code serializes FSImage using in-house serialization > mechanisms. There are a couple disadvantages of the current approach: > # Mixing the responsibility of reconstruction and serialization / > deserialization. The current code paths of serialization / deserialization > have spent a lot of effort on maintaining compatibility. What is worse is > that they are mixed with the complex logic of reconstructing the namespace, > making the code difficult to follow. > # Poor documentation of the current FSImage format. The format of the FSImage > is practically defined by the implementation. An bug in implementation means > a bug in the specification. Furthermore, it also makes writing third-party > tools quite difficult. > # Changing schemas is non-trivial. Adding a field in FSImage requires bumping > the layout version every time. Bumping out layout version requires (1) the > users to explicitly upgrade the clusters, and (2) putting new code to > maintain backward compatibility. > This jira proposes to use protobuf to serialize the FSImage. Protobuf has > been used to serialize / deserialize the RPC message in Hadoop. > Protobuf addresses all the above problems. It clearly separates the > responsibility of serialization and reconstructing the namespace. The > protobuf files document the current format of the FSImage. The developers now > can add optional fields with ease, since the old code can always read the new > FSImage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5698) Use protobuf to serialize / deserialize FSImage
[ https://issues.apache.org/jira/browse/HDFS-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902174#comment-13902174 ] Arpit Agarwal commented on HDFS-5698: - I committed the merge patch to branch-2. Thanks Haohui! > Use protobuf to serialize / deserialize FSImage > --- > > Key: HDFS-5698 > URL: https://issues.apache.org/jira/browse/HDFS-5698 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5698-branch2.000.patch, HDFS-5698-design.pdf, > HDFS-5698.000.patch, HDFS-5698.001.patch, HDFS-5698.002.patch, > HDFS-5698.003.patch, HDFS-5698.004.patch, HDFS-5698.005.patch, > HDFS-5698.006.patch, HDFS-5698.007.patch > > > Currently, the code serializes FSImage using in-house serialization > mechanisms. There are a couple disadvantages of the current approach: > # Mixing the responsibility of reconstruction and serialization / > deserialization. The current code paths of serialization / deserialization > have spent a lot of effort on maintaining compatibility. What is worse is > that they are mixed with the complex logic of reconstructing the namespace, > making the code difficult to follow. > # Poor documentation of the current FSImage format. The format of the FSImage > is practically defined by the implementation. An bug in implementation means > a bug in the specification. Furthermore, it also makes writing third-party > tools quite difficult. > # Changing schemas is non-trivial. Adding a field in FSImage requires bumping > the layout version every time. Bumping out layout version requires (1) the > users to explicitly upgrade the clusters, and (2) putting new code to > maintain backward compatibility. > This jira proposes to use protobuf to serialize the FSImage. Protobuf has > been used to serialize / deserialize the RPC message in Hadoop. > Protobuf addresses all the above problems. It clearly separates the > responsibility of serialization and reconstructing the namespace. The > protobuf files document the current format of the FSImage. The developers now > can add optional fields with ease, since the old code can always read the new > FSImage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5866) '-maxSize' and '-step' option fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-5866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902172#comment-13902172
]
Akira AJISAKA commented on HDFS-5866:
-
bq. Now a file size is multiplied by the replication factor when using
FileDistribution processor.
Created a separate issue (HDFS-5956). I'll split the patch.
> '-maxSize' and '-step' option fail in OfflineImageViewer
>
>
> Key: HDFS-5866
> URL: https://issues.apache.org/jira/browse/HDFS-5866
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-5866.2.patch, HDFS-5866.patch
>
>
> Executing -step or/and -maxSize option will get the following error:
> {code}
> $ hdfs oiv -p FileDistribution -step 102400 -i input -o output
> Error parsing command-line options:
> Usage: bin/hdfs oiv [OPTIONS] -i INPUTFILE -o OUTPUTFILE
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (HDFS-5956) A file size is multiplied by the replication factor in 'hdfs oiv -p FileDistribution' option
Akira AJISAKA created HDFS-5956:
---
Summary: A file size is multiplied by the replication factor in
'hdfs oiv -p FileDistribution' option
Key: HDFS-5956
URL: https://issues.apache.org/jira/browse/HDFS-5956
Project: Hadoop HDFS
Issue Type: Sub-task
Components: tools
Affects Versions: 3.0.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
In FileDistributionCalculator.java,
{code}
long fileSize = 0;
for (BlockProto b : f.getBlocksList()) {
fileSize += b.getNumBytes() * f.getReplication();
}
maxFileSize = Math.max(fileSize, maxFileSize);
totalSpace += fileSize;
{code}
should be
{code}
long fileSize = 0;
for (BlockProto b : f.getBlocksList()) {
fileSize += b.getNumBytes();
}
maxFileSize = Math.max(fileSize, maxFileSize);
totalSpace += fileSize * f.getReplication();
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5716) Allow WebHDFS to use pluggable authentication filter
[
https://issues.apache.org/jira/browse/HDFS-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902156#comment-13902156
]
Brandon Li commented on HDFS-5716:
--
{quote}Here, getFileStatus() should fail or pass?{quote}
It should pass with default filter but fail with the test filter.
I will commit the patch soon.
> Allow WebHDFS to use pluggable authentication filter
>
>
> Key: HDFS-5716
> URL: https://issues.apache.org/jira/browse/HDFS-5716
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5716.000.patch, HDFS-5716.001.patch,
> HDFS-5716.002.patch
>
>
> Currently the WebHDFS server only uses SPNEGO filters for authentication
> purposes. The design serves the purpose well when SPNEGO is available,
> however, other services (e.g., the web UI) where SPNEGO might be unavailable
> can also use WebHDFS.
> Note that the web UI and the WebHDFS are served by the same Jetty server on
> the same port. The Web UI can use customized authentication filters, it might
> make sense to allow WebHDFS to take customized authentication filters as well.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5716) Allow WebHDFS to use pluggable authentication filter
[ https://issues.apache.org/jira/browse/HDFS-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5716: - Component/s: webhdfs > Allow WebHDFS to use pluggable authentication filter > > > Key: HDFS-5716 > URL: https://issues.apache.org/jira/browse/HDFS-5716 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5716.000.patch, HDFS-5716.001.patch, > HDFS-5716.002.patch > > > Currently the WebHDFS server only uses SPNEGO filters for authentication > purposes. The design serves the purpose well when SPNEGO is available, > however, other services (e.g., the web UI) where SPNEGO might be unavailable > can also use WebHDFS. > Note that the web UI and the WebHDFS are served by the same Jetty server on > the same port. The Web UI can use customized authentication filters, it might > make sense to allow WebHDFS to take customized authentication filters as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5866) '-maxSize' and '-step' option fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-5866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-5866:
Attachment: HDFS-5866.2.patch
Add test for these options, and fixed to output actual file size.
Now a file size is multiplied by the replication factor when using
FileDistribution processor.
> '-maxSize' and '-step' option fail in OfflineImageViewer
>
>
> Key: HDFS-5866
> URL: https://issues.apache.org/jira/browse/HDFS-5866
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-5866.2.patch, HDFS-5866.patch
>
>
> Executing -step or/and -maxSize option will get the following error:
> {code}
> $ hdfs oiv -p FileDistribution -step 102400 -i input -o output
> Error parsing command-line options:
> Usage: bin/hdfs oiv [OPTIONS] -i INPUTFILE -o OUTPUTFILE
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5716) Allow WebHDFS to use pluggable authentication filter
[
https://issues.apache.org/jira/browse/HDFS-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902141#comment-13902141
]
Brandon Li commented on HDFS-5716:
--
{noformat}+// getFileStatus() is supposed to pass through with the default
filter.{noformat}
Here, getFileStatus() should fail or pass?
> Allow WebHDFS to use pluggable authentication filter
>
>
> Key: HDFS-5716
> URL: https://issues.apache.org/jira/browse/HDFS-5716
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5716.000.patch, HDFS-5716.001.patch,
> HDFS-5716.002.patch
>
>
> Currently the WebHDFS server only uses SPNEGO filters for authentication
> purposes. The design serves the purpose well when SPNEGO is available,
> however, other services (e.g., the web UI) where SPNEGO might be unavailable
> can also use WebHDFS.
> Note that the web UI and the WebHDFS are served by the same Jetty server on
> the same port. The Web UI can use customized authentication filters, it might
> make sense to allow WebHDFS to take customized authentication filters as well.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5716) Allow WebHDFS to use pluggable authentication filter
[ https://issues.apache.org/jira/browse/HDFS-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5716: - Attachment: HDFS-5716.002.patch Thanks Brandon for the review. The v2 patch addresses Brandon's comment. > Allow WebHDFS to use pluggable authentication filter > > > Key: HDFS-5716 > URL: https://issues.apache.org/jira/browse/HDFS-5716 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5716.000.patch, HDFS-5716.001.patch, > HDFS-5716.002.patch > > > Currently the WebHDFS server only uses SPNEGO filters for authentication > purposes. The design serves the purpose well when SPNEGO is available, > however, other services (e.g., the web UI) where SPNEGO might be unavailable > can also use WebHDFS. > Note that the web UI and the WebHDFS are served by the same Jetty server on > the same port. The Web UI can use customized authentication filters, it might > make sense to allow WebHDFS to take customized authentication filters as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5780) TestRBWBlockInvalidation times out intemittently on branch-2
[
https://issues.apache.org/jira/browse/HDFS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902125#comment-13902125
]
Hadoop QA commented on HDFS-5780:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629113/HDFS-5780.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6157//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6157//console
This message is automatically generated.
> TestRBWBlockInvalidation times out intemittently on branch-2
>
>
> Key: HDFS-5780
> URL: https://issues.apache.org/jira/browse/HDFS-5780
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: HDFS-5780.patch, HDFS-5780.patch
>
>
> i recently found out that the test
> TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN times
> out intermittently.
> I am using Fedora, JDK7
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5920) Support rollback of rolling upgrade in NameNode and JournalNodes
[ https://issues.apache.org/jira/browse/HDFS-5920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902115#comment-13902115 ] Suresh Srinivas commented on HDFS-5920: --- +1 for the patch. > Support rollback of rolling upgrade in NameNode and JournalNodes > > > Key: HDFS-5920 > URL: https://issues.apache.org/jira/browse/HDFS-5920 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: journal-node, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-5920.000.patch, HDFS-5920.000.patch, > HDFS-5920.001.patch, HDFS-5920.002.patch, HDFS-5920.003.patch > > > This jira provides rollback functionality for NameNode and JournalNode in > rolling upgrade. > Currently the proposed rollback for rolling upgrade is: > 1. Shutdown both NN > 2. Start one of the NN using "-rollingUpgrade rollback" option > 3. This NN will load the special fsimage right before the upgrade marker, > then discard all the editlog segments after the txid of the fsimage > 4. The NN will also send RPC requests to all the JNs to discard editlog > segments. This call expects response from all the JNs. The NN will keep > running if the call succeeds. > 5. We start the other NN using bootstrapstandby rather than "-rollingUpgrade > rollback" -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-3405) Checkpointing should use HTTP POST or PUT instead of GET-GET to send merged fsimages
[
https://issues.apache.org/jira/browse/HDFS-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902105#comment-13902105
]
Andrew Wang commented on HDFS-3405:
---
Thanks for the rev Vinay, we're pretty close once we get some more manual
testing done. Some more comments:
General:
* I'd like to not leave the timeout at 10min, and now is actually a good time
to test the timeout config further in depth. Looking at HttpURLConnection, it
has setConnectTimeout and setReadTimeout, but I don't know how that works with
PUT. Namely, I'm worried that the new unit test you added is hitting the
connect or read timeout, rather than slowness/pause during the actual transfer.
I think this isn't that hard to test, since you can combine a low timeout with
a low transfer rate (unit testable), or use Linux's {{tc}} (traffic control) if
you want to get fancy with manual testing. At a high-level, what we want here
is for the NN/SbNN/2NN to not hang indefinitely if the other side goes down
completely, and if this is in fact a socket timeout, then 60s is sufficient. If
it's actually a timeout for the whole transfer, then let's stick with 10m.
* The 2GB+ file test is also pretty important, so let's wait on that as well
before committing.
* Should we refactor out "/imagetransfer" into a static constant in
ImageServlet? We seem to use it a bunch.
ImageServlet:
* Class javadoc should mention the Standby NameNode in addition to the
Secondary NameNode, since that's actually the more common deployment with
modern Hadoop.
TransferFsImage:
* "Uploades" is still spelled wrong
* Reference to "/putimage" in a comment needs to be updated
* We should be throttling on the receiver side, not the sender. This way both
GETs from and PUTs to a NN will use the same throttle setting.
TestTransferFsImage:
* Let's not create a temp file in a global directory, I just had to fix some
problems like this in HDFS-3128. Instead, you can do {{FileSystem.getLocal}}
and pass that to {{FileSystemTestHelper#getTestRootPath}} to make a safe unique
temp file.
* Comment explicitly mentions default 10min timeout, let's just say "otherwise
it will wait for the default" instead since we might change the default
> Checkpointing should use HTTP POST or PUT instead of GET-GET to send merged
> fsimages
>
>
> Key: HDFS-3405
> URL: https://issues.apache.org/jira/browse/HDFS-3405
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 1.0.0, 3.0.0, 2.0.5-alpha
>Reporter: Aaron T. Myers
>Assignee: Vinayakumar B
> Attachments: HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch,
> HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch,
> HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch,
> HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch
>
>
> As Todd points out in [this
> comment|https://issues.apache.org/jira/browse/HDFS-3404?focusedCommentId=13272986&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13272986],
> the current scheme for a checkpointing daemon to upload a merged fsimage
> file to an NN is to issue an HTTP get request to tell the target NN to issue
> another GET request back to the checkpointing daemon to retrieve the merged
> fsimage file. There's no fundamental reason the checkpointing daemon can't
> just use an HTTP POST or PUT to send back the merged fsimage file, rather
> than the double-GET scheme.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902095#comment-13902095 ] stack commented on HDFS-5274: - [~iwasakims] I'm pretty sure Elliott would be happy if you took it. I can help (not sure if that a hinderance or a help -- smile). > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, > Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5716) Allow WebHDFS to use pluggable authentication filter
[ https://issues.apache.org/jira/browse/HDFS-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902083#comment-13902083 ] Brandon Li commented on HDFS-5716: -- +1. The patch looks good to me. Minor: it would be nice to add some comments to the unit test testWebHdfsAuthFilter(). > Allow WebHDFS to use pluggable authentication filter > > > Key: HDFS-5716 > URL: https://issues.apache.org/jira/browse/HDFS-5716 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5716.000.patch, HDFS-5716.001.patch > > > Currently the WebHDFS server only uses SPNEGO filters for authentication > purposes. The design serves the purpose well when SPNEGO is available, > however, other services (e.g., the web UI) where SPNEGO might be unavailable > can also use WebHDFS. > Note that the web UI and the WebHDFS are served by the same Jetty server on > the same port. The Web UI can use customized authentication filters, it might > make sense to allow WebHDFS to take customized authentication filters as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5955) branch-2 fails to compile
Arpit Agarwal created HDFS-5955:
---
Summary: branch-2 fails to compile
Key: HDFS-5955
URL: https://issues.apache.org/jira/browse/HDFS-5955
Project: Hadoop HDFS
Issue Type: Bug
Components: build
Affects Versions: 2.4.0
Reporter: Arpit Agarwal
Priority: Critical
I get the following error compiling branch-2.
{code}
Picked up _JAVA_OPTIONS: -Djava.awt.headless=true
[ERROR] COMPILATION ERROR :
[ERROR]
/Users/aagarwal/src/hdp2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer.java:[223,20]
cannot find symbol
symbol : method isSecure()
location: class org.apache.hadoop.http.HttpConfig
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile)
on project hadoop-common: Compilation failure
[ERROR]
/Users/aagarwal/src/hdp2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer.java:[223,20]
cannot find symbol
[ERROR] symbol : method isSecure()
[ERROR] location: class org.apache.hadoop.http.HttpConfig
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5949) New Namenode UI when trying to download a file, the browser doesn't know the file name
[
https://issues.apache.org/jira/browse/HDFS-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902059#comment-13902059
]
Hudson commented on HDFS-5949:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5174 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5174/])
HDFS-5949. New Namenode UI when trying to download a file, the browser doesn't
know the file name.Contributed by Haohui Mai (brandonli:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568536)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/explorer.js
> New Namenode UI when trying to download a file, the browser doesn't know the
> file name
> --
>
> Key: HDFS-5949
> URL: https://issues.apache.org/jira/browse/HDFS-5949
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Travis Thompson
>Assignee: Travis Thompson
>Priority: Minor
> Fix For: 2.4.0
>
> Attachments: HDFS-5949-1.patch
>
>
> When trying to download a file though the new Namenode UI FS Browser, the
> browser doesn't know the name of the file because of a trailing slash. For
> instance, this url is broken and Firefox picks a random name for:
> {noformat}
> http://dn.example.com:70/webhdfs/v1/user/tthompso/test_examples/wordcount_in/core-site.xml/?op=OPEN&delegation=TOKEN&namenoderpcaddress=namenode.example.com:9000&offset=0
> {noformat}
> But if you remove the trailing / on the file name, Firefox correctly picks up
> the name of the file:
> {noformat}
> http://dn.example.com:70/webhdfs/v1/user/tthompso/test_examples/wordcount_in/core-site.xml?op=OPEN&delegation=TOKEN&namenoderpcaddress=namenode.example.com:9000&offset=0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HDFS-5776: Attachment: HDFS-5776v18.txt Make minimal changes to address @Jing Zhao review comments (the first one is a good catch). > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v10.txt, HDFS-5776-v11.txt, HDFS-5776-v12.txt, > HDFS-5776-v12.txt, HDFS-5776-v13.wip.txt, HDFS-5776-v14.txt, > HDFS-5776-v15.txt, HDFS-5776-v17.txt, HDFS-5776-v17.txt, HDFS-5776-v2.txt, > HDFS-5776-v3.txt, HDFS-5776-v4.txt, HDFS-5776-v5.txt, HDFS-5776-v6.txt, > HDFS-5776-v7.txt, HDFS-5776-v8.txt, HDFS-5776-v9.txt, HDFS-5776.txt, > HDFS-5776v18.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5949) New Namenode UI when trying to download a file, the browser doesn't know the file name
[
https://issues.apache.org/jira/browse/HDFS-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-5949:
-
Component/s: namenode
> New Namenode UI when trying to download a file, the browser doesn't know the
> file name
> --
>
> Key: HDFS-5949
> URL: https://issues.apache.org/jira/browse/HDFS-5949
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Travis Thompson
>Assignee: Travis Thompson
>Priority: Minor
> Fix For: 2.4.0
>
> Attachments: HDFS-5949-1.patch
>
>
> When trying to download a file though the new Namenode UI FS Browser, the
> browser doesn't know the name of the file because of a trailing slash. For
> instance, this url is broken and Firefox picks a random name for:
> {noformat}
> http://dn.example.com:70/webhdfs/v1/user/tthompso/test_examples/wordcount_in/core-site.xml/?op=OPEN&delegation=TOKEN&namenoderpcaddress=namenode.example.com:9000&offset=0
> {noformat}
> But if you remove the trailing / on the file name, Firefox correctly picks up
> the name of the file:
> {noformat}
> http://dn.example.com:70/webhdfs/v1/user/tthompso/test_examples/wordcount_in/core-site.xml?op=OPEN&delegation=TOKEN&namenoderpcaddress=namenode.example.com:9000&offset=0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5951) Provide diagnosis information in the Web UI
[ https://issues.apache.org/jira/browse/HDFS-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902048#comment-13902048 ] Todd Lipcon commented on HDFS-5951: --- I agree with Aaron. I can think of several good reasons against self-monitoring systems: - It is impossible to check for things like external network connectivity. For example, if a NN sees that it is getting 0 requests/sec, that may indicate that the network is down, or it may just indicate that there are no clients. An external system can provide much better data by actually checking that the NN is accessible and correctly functioning (eg a canary) - Similarly, if the RPC subsystem is dead, we can't tell that internally - we need something like an external canary to tell us - In my experience, a large majority of issues we see in HDFS are due to some environmental issues -- for example frame errors on the NIC, machine swapping, underprovisioned network resources, failing HDs, etc. These are obviously out-of-scope for the NN to monitor, right? Given that any competent operator needs to monitor all of the above, do they really gain a lot by also having a web UI notice? Additionally, a useful monitoring system has a lot more than a simple notice on a web page. For example: - SNMP traps to notify external systems of issues (bubble-up to corporate NOC for example) - Email or other alerts for issues. - Configurable thresholds for metrics-based checks - Historical information available to triggers (eg "metric X is above value Y for at least Z minutes in a row") I think we'll all agree that the above are out of scope for a system like HDFS. Instead, HDFS should make sure that all interesting data is exposed as metrics, and that the metrics are documented (perhaps with some advice on thresholds). Additionally the community might make available a set of scripts to poll the metrics which could be hooked into external systems like Nagios, etc. > Provide diagnosis information in the Web UI > --- > > Key: HDFS-5951 > URL: https://issues.apache.org/jira/browse/HDFS-5951 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5951.000.patch, diagnosis-failure.png, > diagnosis-succeed.png > > > HDFS should provide operation statistics in its UI. it can go one step > further by leveraging the information to diagnose common problems. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5949) New Namenode UI when trying to download a file, the browser doesn't know the file name
[
https://issues.apache.org/jira/browse/HDFS-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-5949:
-
Fix Version/s: 2.4.0
> New Namenode UI when trying to download a file, the browser doesn't know the
> file name
> --
>
> Key: HDFS-5949
> URL: https://issues.apache.org/jira/browse/HDFS-5949
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Travis Thompson
>Assignee: Travis Thompson
>Priority: Minor
> Fix For: 2.4.0
>
> Attachments: HDFS-5949-1.patch
>
>
> When trying to download a file though the new Namenode UI FS Browser, the
> browser doesn't know the name of the file because of a trailing slash. For
> instance, this url is broken and Firefox picks a random name for:
> {noformat}
> http://dn.example.com:70/webhdfs/v1/user/tthompso/test_examples/wordcount_in/core-site.xml/?op=OPEN&delegation=TOKEN&namenoderpcaddress=namenode.example.com:9000&offset=0
> {noformat}
> But if you remove the trailing / on the file name, Firefox correctly picks up
> the name of the file:
> {noformat}
> http://dn.example.com:70/webhdfs/v1/user/tthompso/test_examples/wordcount_in/core-site.xml?op=OPEN&delegation=TOKEN&namenoderpcaddress=namenode.example.com:9000&offset=0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5949) New Namenode UI when trying to download a file, the browser doesn't know the file name
[
https://issues.apache.org/jira/browse/HDFS-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-5949:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> New Namenode UI when trying to download a file, the browser doesn't know the
> file name
> --
>
> Key: HDFS-5949
> URL: https://issues.apache.org/jira/browse/HDFS-5949
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.3.0
>Reporter: Travis Thompson
>Assignee: Travis Thompson
>Priority: Minor
> Attachments: HDFS-5949-1.patch
>
>
> When trying to download a file though the new Namenode UI FS Browser, the
> browser doesn't know the name of the file because of a trailing slash. For
> instance, this url is broken and Firefox picks a random name for:
> {noformat}
> http://dn.example.com:70/webhdfs/v1/user/tthompso/test_examples/wordcount_in/core-site.xml/?op=OPEN&delegation=TOKEN&namenoderpcaddress=namenode.example.com:9000&offset=0
> {noformat}
> But if you remove the trailing / on the file name, Firefox correctly picks up
> the name of the file:
> {noformat}
> http://dn.example.com:70/webhdfs/v1/user/tthompso/test_examples/wordcount_in/core-site.xml?op=OPEN&delegation=TOKEN&namenoderpcaddress=namenode.example.com:9000&offset=0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5780) TestRBWBlockInvalidation times out intemittently on branch-2
[ https://issues.apache.org/jira/browse/HDFS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated HDFS-5780: Status: Patch Available (was: Open) > TestRBWBlockInvalidation times out intemittently on branch-2 > > > Key: HDFS-5780 > URL: https://issues.apache.org/jira/browse/HDFS-5780 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0, 3.0.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: HDFS-5780.patch, HDFS-5780.patch > > > i recently found out that the test > TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN times > out intermittently. > I am using Fedora, JDK7 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902036#comment-13902036
]
Hadoop QA commented on HDFS-5274:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12607415/HDFS-5274-6.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6158//console
This message is automatically generated.
> Add Tracing to HDFS
> ---
>
> Key: HDFS-5274
> URL: https://issues.apache.org/jira/browse/HDFS-5274
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, namenode
>Affects Versions: 2.1.1-beta
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
> HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
> Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png
>
>
> Since Google's Dapper paper has shown the benefits of tracing for a large
> distributed system, it seems like a good time to add tracing to HDFS. HBase
> has added tracing using HTrace. I propose that the same can be done within
> HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5780) TestRBWBlockInvalidation times out intemittently on branch-2
[ https://issues.apache.org/jira/browse/HDFS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated HDFS-5780: Attachment: HDFS-5780.patch Attaching the new patch with the addressed changes. I have increased the timeout to 10minutes and I had to make few other timing related changes. > TestRBWBlockInvalidation times out intemittently on branch-2 > > > Key: HDFS-5780 > URL: https://issues.apache.org/jira/browse/HDFS-5780 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: HDFS-5780.patch, HDFS-5780.patch > > > i recently found out that the test > TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN times > out intermittently. > I am using Fedora, JDK7 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[ https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902025#comment-13902025 ] Masatake Iwasaki commented on HDFS-5274: Hi [~eclark], Are you working on this issue now? If you do not have enough time, I would like to help fixing and rebasing this patch. > Add Tracing to HDFS > --- > > Key: HDFS-5274 > URL: https://issues.apache.org/jira/browse/HDFS-5274 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 2.1.1-beta >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch, > HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch, > Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png > > > Since Google's Dapper paper has shown the benefits of tracing for a large > distributed system, it seems like a good time to add tracing to HDFS. HBase > has added tracing using HTrace. I propose that the same can be done within > HDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5949) New Namenode UI when trying to download a file, the browser doesn't know the file name
[
https://issues.apache.org/jira/browse/HDFS-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902026#comment-13902026
]
Brandon Li commented on HDFS-5949:
--
I've committed the patch. Thank you, Travis and Haohui.
> New Namenode UI when trying to download a file, the browser doesn't know the
> file name
> --
>
> Key: HDFS-5949
> URL: https://issues.apache.org/jira/browse/HDFS-5949
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.3.0
>Reporter: Travis Thompson
>Assignee: Travis Thompson
>Priority: Minor
> Attachments: HDFS-5949-1.patch
>
>
> When trying to download a file though the new Namenode UI FS Browser, the
> browser doesn't know the name of the file because of a trailing slash. For
> instance, this url is broken and Firefox picks a random name for:
> {noformat}
> http://dn.example.com:70/webhdfs/v1/user/tthompso/test_examples/wordcount_in/core-site.xml/?op=OPEN&delegation=TOKEN&namenoderpcaddress=namenode.example.com:9000&offset=0
> {noformat}
> But if you remove the trailing / on the file name, Firefox correctly picks up
> the name of the file:
> {noformat}
> http://dn.example.com:70/webhdfs/v1/user/tthompso/test_examples/wordcount_in/core-site.xml?op=OPEN&delegation=TOKEN&namenoderpcaddress=namenode.example.com:9000&offset=0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5951) Provide diagnosis information in the Web UI
[ https://issues.apache.org/jira/browse/HDFS-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902022#comment-13902022 ] Suresh Srinivas commented on HDFS-5951: --- bq. Hey guys, I'm not really sure this is the sort of thing that should be in HDFS at all. Doesn't Apache Ambari already do stuff like this? I think there are lot of folks who use Hadoop as is. This is not to provide super sophisticated UI, but to flag obvious errors. This helps in debugging and for people to understand issues in the cluster. Can you explain the reasons for why this should not be in HDFS? > Provide diagnosis information in the Web UI > --- > > Key: HDFS-5951 > URL: https://issues.apache.org/jira/browse/HDFS-5951 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5951.000.patch, diagnosis-failure.png, > diagnosis-succeed.png > > > HDFS should provide operation statistics in its UI. it can go one step > further by leveraging the information to diagnose common problems. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5948) TestBackupNode flakes with port in use error
[
https://issues.apache.org/jira/browse/HDFS-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902020#comment-13902020
]
Hudson commented on HDFS-5948:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5173 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5173/])
HDFS-5948. TestBackupNode flakes with port in use error. (Contributed by Haohui
Mai) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568532)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestBackupNode.java
> TestBackupNode flakes with port in use error
>
>
> Key: HDFS-5948
> URL: https://issues.apache.org/jira/browse/HDFS-5948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Haohui Mai
> Fix For: 2.4.0
>
> Attachments: HDFS-5948.000.patch
>
>
> TestBackupNode has flaked on our internal Jenkins and I found it in a few
> test-patch runs upstream too. Error like this:
> {noformat}
> Error Message:
> Port in use: 0.0.0.0:50105
> Stack Trace:
> java.lang.AssertionError: Port in use: 0.0.0.0:50105
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:295)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpointNode(TestBackupNode.java:124)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5951) Provide diagnosis information in the Web UI
[ https://issues.apache.org/jira/browse/HDFS-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13902014#comment-13902014 ] Haohui Mai commented on HDFS-5951: -- I think Ambari is more focused on providing statistics, but the point of the work is to pass down the experience from operation / support practice as rules so that the wisdom can benefit the general users. For example, the current patch implements a rule that issues a warning if the namenode has not been checkpointed for a while. We plan to implement checks on whether the size of heap is appropriate for the scale of the namespace, etc. These rules are not necessarily monitoring rules, they provide suggestions on best practice on operation. It seems to me that they are highly specific to HDFS. It should be the responsibility of the HDFS developers to provide these rules and let external entities (e.g., Ambari) to query the results from HDFS. > Provide diagnosis information in the Web UI > --- > > Key: HDFS-5951 > URL: https://issues.apache.org/jira/browse/HDFS-5951 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5951.000.patch, diagnosis-failure.png, > diagnosis-succeed.png > > > HDFS should provide operation statistics in its UI. it can go one step > further by leveraging the information to diagnose common problems. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5948) TestBackupNode flakes with port in use error
[
https://issues.apache.org/jira/browse/HDFS-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5948:
Resolution: Fixed
Fix Version/s: 2.4.0
Target Version/s: 2.4.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk and branch-2. Thanks for the contribution [~wheat9].
> TestBackupNode flakes with port in use error
>
>
> Key: HDFS-5948
> URL: https://issues.apache.org/jira/browse/HDFS-5948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Haohui Mai
> Fix For: 2.4.0
>
> Attachments: HDFS-5948.000.patch
>
>
> TestBackupNode has flaked on our internal Jenkins and I found it in a few
> test-patch runs upstream too. Error like this:
> {noformat}
> Error Message:
> Port in use: 0.0.0.0:50105
> Stack Trace:
> java.lang.AssertionError: Port in use: 0.0.0.0:50105
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:295)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpointNode(TestBackupNode.java:124)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5780) TestRBWBlockInvalidation times out intemittently on branch-2
[ https://issues.apache.org/jira/browse/HDFS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated HDFS-5780: Status: Open (was: Patch Available) > TestRBWBlockInvalidation times out intemittently on branch-2 > > > Key: HDFS-5780 > URL: https://issues.apache.org/jira/browse/HDFS-5780 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0, 3.0.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: HDFS-5780.patch > > > i recently found out that the test > TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN times > out intermittently. > I am using Fedora, JDK7 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5948) TestBackupNode flakes with port in use error
[
https://issues.apache.org/jira/browse/HDFS-5948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901989#comment-13901989
]
Arpit Agarwal commented on HDFS-5948:
-
+1 for the patch, nice fix!
> TestBackupNode flakes with port in use error
>
>
> Key: HDFS-5948
> URL: https://issues.apache.org/jira/browse/HDFS-5948
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Andrew Wang
>Assignee: Haohui Mai
> Attachments: HDFS-5948.000.patch
>
>
> TestBackupNode has flaked on our internal Jenkins and I found it in a few
> test-patch runs upstream too. Error like this:
> {noformat}
> Error Message:
> Port in use: 0.0.0.0:50105
> Stack Trace:
> java.lang.AssertionError: Port in use: 0.0.0.0:50105
> at org.junit.Assert.fail(Assert.java:91)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:295)
> at
> org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpointNode(TestBackupNode.java:124)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-3128) Unit tests should not use a test root in /tmp
[ https://issues.apache.org/jira/browse/HDFS-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901978#comment-13901978 ] Hudson commented on HDFS-3128: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5172 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5172/]) HDFS-3128. Unit tests should not use a test root in /tmp. (wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568526) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FSMainOperationsBaseTest.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileContextTestHelper.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FileSystemTestHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsCreateMkdir.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsPermission.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestFcHdfsSetUMask.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestHDFSFileContextMainOperations.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestSymlinkHdfsDisable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemAtHdfsRoot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemHdfs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsAtHdfsRoot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFsHdfs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestFSMainOperationsWebHdfs.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/PathUtils.java > Unit tests should not use a test root in /tmp > - > > Key: HDFS-3128 > URL: https://issues.apache.org/jira/browse/HDFS-3128 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.3.0 >Reporter: Eli Collins >Assignee: Andrew Wang >Priority: Minor > Fix For: 2.4.0 > > Attachments: hdfs-3128-1.patch > > > Saw this on jenkins, TestResolveHdfsSymlink#testFcResolveAfs creates > /tmp/alpha which interferes with other executors on the same machine. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5780) TestRBWBlockInvalidation times out intemittently on branch-2
[ https://issues.apache.org/jira/browse/HDFS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901969#comment-13901969 ] Mit Desai commented on HDFS-5780: - Thanks Arpit. I will address your concerns and post another patch. > TestRBWBlockInvalidation times out intemittently on branch-2 > > > Key: HDFS-5780 > URL: https://issues.apache.org/jira/browse/HDFS-5780 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: HDFS-5780.patch > > > i recently found out that the test > TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN times > out intermittently. > I am using Fedora, JDK7 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-3128) Unit tests should not use a test root in /tmp
[ https://issues.apache.org/jira/browse/HDFS-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-3128: -- Resolution: Fixed Fix Version/s: 2.4.0 Status: Resolved (was: Patch Available) Thanks for reviewing, Colin and Yongjun. I committed this to trunk and branch-2. > Unit tests should not use a test root in /tmp > - > > Key: HDFS-3128 > URL: https://issues.apache.org/jira/browse/HDFS-3128 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.3.0 >Reporter: Eli Collins >Assignee: Andrew Wang >Priority: Minor > Fix For: 2.4.0 > > Attachments: hdfs-3128-1.patch > > > Saw this on jenkins, TestResolveHdfsSymlink#testFcResolveAfs creates > /tmp/alpha which interferes with other executors on the same machine. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-3128) Unit tests should not use a test root in /tmp
[ https://issues.apache.org/jira/browse/HDFS-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-3128: -- Summary: Unit tests should not use a test root in /tmp (was: TestResolveHdfsSymlink#testFcResolveAfs shouldn't use /tmp) > Unit tests should not use a test root in /tmp > - > > Key: HDFS-3128 > URL: https://issues.apache.org/jira/browse/HDFS-3128 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.3.0 >Reporter: Eli Collins >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-3128-1.patch > > > Saw this on jenkins, TestResolveHdfsSymlink#testFcResolveAfs creates > /tmp/alpha which interferes with other executors on the same machine. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-3128) TestResolveHdfsSymlink#testFcResolveAfs shouldn't use /tmp
[ https://issues.apache.org/jira/browse/HDFS-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901953#comment-13901953 ] Yongjun Zhang commented on HDFS-3128: - +1. Thanks Andrew. Good fix! I went through the change, and I searched in the whole code base, just to see if anything is missed, I didn't catch any. One thing I noticed is that, not because of your change, there are lots of places in test area and one place not in test area (DBCountPageView.java) that hard-code the property string "test.build.data" . Probably we can consider adding a constant to common property file for this and replace them all with the constant as a future improvement. Thanks. > TestResolveHdfsSymlink#testFcResolveAfs shouldn't use /tmp > -- > > Key: HDFS-3128 > URL: https://issues.apache.org/jira/browse/HDFS-3128 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.3.0 >Reporter: Eli Collins >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-3128-1.patch > > > Saw this on jenkins, TestResolveHdfsSymlink#testFcResolveAfs creates > /tmp/alpha which interferes with other executors on the same machine. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HDFS-5585) Provide admin commands for data node upgrade
[ https://issues.apache.org/jira/browse/HDFS-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-5585. -- Resolution: Fixed Fix Version/s: HDFS-5535 (Rolling upgrades) Hadoop Flags: Reviewed Thanks for the review, Vinay and Brandon. I've committed this to the RU branch. > Provide admin commands for data node upgrade > > > Key: HDFS-5585 > URL: https://issues.apache.org/jira/browse/HDFS-5585 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Fix For: HDFS-5535 (Rolling upgrades) > > Attachments: HDFS-5585.patch, HDFS-5585.patch, HDFS-5585.patch > > > Several new methods to ClientDatanodeProtocol may need to be added to support > querying version, initiating upgrade, etc. The admin CLI needs to be added > as well. This primary use case is for rolling upgrade, but this can be used > for preparing for a graceful restart of a data node for any reasons. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5898) Allow NFS gateway to login/relogin from its kerberos keytab
[
https://issues.apache.org/jira/browse/HDFS-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901949#comment-13901949
]
Hadoop QA commented on HDFS-5898:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629112/HDFS-5898.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-nfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6156//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6156//console
This message is automatically generated.
> Allow NFS gateway to login/relogin from its kerberos keytab
> ---
>
> Key: HDFS-5898
> URL: https://issues.apache.org/jira/browse/HDFS-5898
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Affects Versions: 2.2.0, 2.4.0
>Reporter: Jing Zhao
>Assignee: Abin Shahab
> Attachments: HDFS-5898-documentation.patch,
> HDFS-5898-documentation.patch, HDFS-5898.patch, HDFS-5898.patch,
> HDFS-5898.patch
>
>
> According to the discussion in HDFS-5804:
> 1. The NFS gateway should be able to get it's own tgts, and renew them.
> 2. We should update the HdfsNfsGateway.apt.vm
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5780) TestRBWBlockInvalidation times out intemittently on branch-2
[
https://issues.apache.org/jira/browse/HDFS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901946#comment-13901946
]
Arpit Agarwal commented on HDFS-5780:
-
Thanks for reporting and fixing this [~mitdesai].
If 60 seconds was too short as the test timeout I suggest increasing it to 5 or
10 minutes instead of just 70 seconds. It is okay for these timeouts to be
conservative.
Coding style nitpicks:
# There should be spaces around the = sign in {{liveReplicas=countReplicas}}
# Spaces around the + sign in {{"+liveReplicas}}.
# Space after the {{//}} for comments.
The patch looks good otherwise.
> TestRBWBlockInvalidation times out intemittently on branch-2
>
>
> Key: HDFS-5780
> URL: https://issues.apache.org/jira/browse/HDFS-5780
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Mit Desai
>Assignee: Mit Desai
> Attachments: HDFS-5780.patch
>
>
> i recently found out that the test
> TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN times
> out intermittently.
> I am using Fedora, JDK7
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Resolved] (HDFS-5954) Merge Protobuf-based-FSImage code from trunk
[ https://issues.apache.org/jira/browse/HDFS-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-5954. - Resolution: Fixed Fix Version/s: HDFS-5535 (Rolling upgrades) Target Version/s: HDFS-5535 (Rolling upgrades) Hadoop Flags: Reviewed +1 for the patch. I committed it to branch HDFS-5535. Thanks for taking care of this Jing! > Merge Protobuf-based-FSImage code from trunk > > > Key: HDFS-5954 > URL: https://issues.apache.org/jira/browse/HDFS-5954 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Fix For: HDFS-5535 (Rolling upgrades) > > Attachments: HDFS-5954.patch > > > After merging the protobuf-based-fsimage code from trunk, we need to fix some > compilation errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5780) TestRBWBlockInvalidation times out intemittently on branch-2
[ https://issues.apache.org/jira/browse/HDFS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated HDFS-5780: Attachment: HDFS-5780.patch Attaching the patch. We need to change the conditions in the test because the test failure is due to the Replication Monitor coming and making the changes to the corrupted block before the test checks for it. The test will than keep on waiting for the change to happen. > TestRBWBlockInvalidation times out intemittently on branch-2 > > > Key: HDFS-5780 > URL: https://issues.apache.org/jira/browse/HDFS-5780 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.2.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: HDFS-5780.patch > > > i recently found out that the test > TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN times > out intermittently. > I am using Fedora, JDK7 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5780) TestRBWBlockInvalidation times out intemittently on branch-2
[ https://issues.apache.org/jira/browse/HDFS-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated HDFS-5780: Affects Version/s: 3.0.0 Status: Patch Available (was: Open) > TestRBWBlockInvalidation times out intemittently on branch-2 > > > Key: HDFS-5780 > URL: https://issues.apache.org/jira/browse/HDFS-5780 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.2.0, 3.0.0 >Reporter: Mit Desai >Assignee: Mit Desai > Attachments: HDFS-5780.patch > > > i recently found out that the test > TestRBWBlockInvalidation#testBlockInvalidationWhenRBWReplicaMissedInDN times > out intermittently. > I am using Fedora, JDK7 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5898) Allow NFS gateway to login/relogin from its kerberos keytab
[ https://issues.apache.org/jira/browse/HDFS-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abin Shahab updated HDFS-5898: -- Attachment: HDFS-5898.patch Fixed build issue. > Allow NFS gateway to login/relogin from its kerberos keytab > --- > > Key: HDFS-5898 > URL: https://issues.apache.org/jira/browse/HDFS-5898 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 2.2.0, 2.4.0 >Reporter: Jing Zhao >Assignee: Abin Shahab > Attachments: HDFS-5898-documentation.patch, > HDFS-5898-documentation.patch, HDFS-5898.patch, HDFS-5898.patch, > HDFS-5898.patch > > > According to the discussion in HDFS-5804: > 1. The NFS gateway should be able to get it's own tgts, and renew them. > 2. We should update the HdfsNfsGateway.apt.vm -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5953) TestBlockReaderFactory fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901891#comment-13901891
]
Hadoop QA commented on HDFS-5953:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629065/HDFS-5953.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6154//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6154//console
This message is automatically generated.
> TestBlockReaderFactory fails in trunk
> -
>
> Key: HDFS-5953
> URL: https://issues.apache.org/jira/browse/HDFS-5953
> Project: Hadoop HDFS
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Akira AJISAKA
> Attachments: HDFS-5953.patch
>
>
> From
> https://builds.apache.org/job/Hadoop-Hdfs-trunk/1673/testReport/junit/org.apache.hadoop.hdfs/TestBlockReaderFactory/testFallbackFromShortCircuitToUnixDomainTraffic/
> :
> {code}
> java.lang.RuntimeException: Although a UNIX domain socket path is configured
> as
> /tmp/socks.1392383436573.1418778351/testFallbackFromShortCircuitToUnixDomainTraffic._PORT,
> we cannot start a localDataXceiverServer because libhadoop cannot be loaded.
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.getDomainPeerServer(DataNode.java:601)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.initDataXceiver(DataNode.java:573)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:769)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:315)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1864)
> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1764)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1243)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:699)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:359)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:340)
> at
> org.apache.hadoop.hdfs.TestBlockReaderFactory.testFallbackFromShortCircuitToUnixDomainTraffic(TestBlockReaderFactory.java:99)
> {code}
> This test failure can be reproduced locally (on Mac).
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5898) Allow NFS gateway to login/relogin from its kerberos keytab
[
https://issues.apache.org/jira/browse/HDFS-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901882#comment-13901882
]
Hadoop QA commented on HDFS-5898:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629097/HDFS-5898.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6155//console
This message is automatically generated.
> Allow NFS gateway to login/relogin from its kerberos keytab
> ---
>
> Key: HDFS-5898
> URL: https://issues.apache.org/jira/browse/HDFS-5898
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Affects Versions: 2.2.0, 2.4.0
>Reporter: Jing Zhao
>Assignee: Abin Shahab
> Attachments: HDFS-5898-documentation.patch,
> HDFS-5898-documentation.patch, HDFS-5898.patch, HDFS-5898.patch
>
>
> According to the discussion in HDFS-5804:
> 1. The NFS gateway should be able to get it's own tgts, and renew them.
> 2. We should update the HdfsNfsGateway.apt.vm
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5951) Provide diagnosis information in the Web UI
[ https://issues.apache.org/jira/browse/HDFS-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901879#comment-13901879 ] Aaron T. Myers commented on HDFS-5951: -- Hey guys, I'm not really sure this is the sort of thing that should be in HDFS at all. Doesn't Apache Ambari already do stuff like this? > Provide diagnosis information in the Web UI > --- > > Key: HDFS-5951 > URL: https://issues.apache.org/jira/browse/HDFS-5951 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5951.000.patch, diagnosis-failure.png, > diagnosis-succeed.png > > > HDFS should provide operation statistics in its UI. it can go one step > further by leveraging the information to diagnose common problems. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5951) Provide diagnosis information in the Web UI
[ https://issues.apache.org/jira/browse/HDFS-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901868#comment-13901868 ] Brandon Li commented on HDFS-5951: -- It's nice to have checkpoint status available on UI. > Provide diagnosis information in the Web UI > --- > > Key: HDFS-5951 > URL: https://issues.apache.org/jira/browse/HDFS-5951 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5951.000.patch, diagnosis-failure.png, > diagnosis-succeed.png > > > HDFS should provide operation statistics in its UI. it can go one step > further by leveraging the information to diagnose common problems. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5951) Provide diagnosis information in the Web UI
[ https://issues.apache.org/jira/browse/HDFS-5951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901872#comment-13901872 ] Brandon Li commented on HDFS-5951: -- +1 > Provide diagnosis information in the Web UI > --- > > Key: HDFS-5951 > URL: https://issues.apache.org/jira/browse/HDFS-5951 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5951.000.patch, diagnosis-failure.png, > diagnosis-succeed.png > > > HDFS should provide operation statistics in its UI. it can go one step > further by leveraging the information to diagnose common problems. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-3128) TestResolveHdfsSymlink#testFcResolveAfs shouldn't use /tmp
[ https://issues.apache.org/jira/browse/HDFS-3128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901865#comment-13901865 ] Colin Patrick McCabe commented on HDFS-3128: +1. Thanks, Andrew. > TestResolveHdfsSymlink#testFcResolveAfs shouldn't use /tmp > -- > > Key: HDFS-3128 > URL: https://issues.apache.org/jira/browse/HDFS-3128 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.3.0 >Reporter: Eli Collins >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-3128-1.patch > > > Saw this on jenkins, TestResolveHdfsSymlink#testFcResolveAfs creates > /tmp/alpha which interferes with other executors on the same machine. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5898) Allow NFS gateway to login/relogin from its kerberos keytab
[ https://issues.apache.org/jira/browse/HDFS-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abin Shahab updated HDFS-5898: -- Attachment: HDFS-5898.patch Added test. > Allow NFS gateway to login/relogin from its kerberos keytab > --- > > Key: HDFS-5898 > URL: https://issues.apache.org/jira/browse/HDFS-5898 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 2.2.0, 2.4.0 >Reporter: Jing Zhao >Assignee: Abin Shahab > Attachments: HDFS-5898-documentation.patch, > HDFS-5898-documentation.patch, HDFS-5898.patch, HDFS-5898.patch > > > According to the discussion in HDFS-5804: > 1. The NFS gateway should be able to get it's own tgts, and renew them. > 2. We should update the HdfsNfsGateway.apt.vm -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5954) Merge Protobuf-based-FSImage code from trunk
[ https://issues.apache.org/jira/browse/HDFS-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901854#comment-13901854 ] Haohui Mai commented on HDFS-5954: -- Yes. Sorry about the confusion. > Merge Protobuf-based-FSImage code from trunk > > > Key: HDFS-5954 > URL: https://issues.apache.org/jira/browse/HDFS-5954 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-5954.patch > > > After merging the protobuf-based-fsimage code from trunk, we need to fix some > compilation errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-4239) Means of telling the datanode to stop using a sick disk
[
https://issues.apache.org/jira/browse/HDFS-4239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901848#comment-13901848
]
Yongjun Zhang commented on HDFS-4239:
-
HI Jimmy,
Thanks for the good work. I went through patch v4. It looks good to me. I only
had a few comments, mostly cosmetic things and I may be wrong myself.
1. In DataNode.java:
private void checkSuperuserPrivilege(String method) throws IOException {
if (checkKerberosAuthMethod(method)) {
...
}
}
The above function check super privilege only when kerberos authentication
is enabled. This seems not restrictive enough to me. However, I saw existing
code in same file also does that, such as:
private void checkBlockLocalPathAccess() throws IOException {
checkKerberosAuthMethod("getBlockLocalPathInfo()");
...
}
So I'm not actually not sure. Please correct me if I'm wrong. Say, I found some
other existing code that checks superuser privilege like
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
public void checkSuperuserPrivilege()
Which seems to do thing differently.
2. In DataNode.java:
/** Ensure that authentication method is kerberos */
boolean checkKerberosAuthMethod(String msg) throws IOException {
Suggest to change (both comments and interface) to something like:
/** Check whether authentication method is kerberos, return true
* if so and false otherwise
*/
boolean isKerberosAuthMethodEnabled(...)...
3. In BlockPoolSliceScanner.java
private static final String VERIFICATION_PREFIX =
"dncp_block_verification.log";
You removed "private" from the interface, I wonder if it's what you intended.
Seems it should stay private.
4. In DatablockScanner.java:
void volumeMarkedDown(FsVolumeSpi vol) throws IOException {
I wonder whether if we can change it to
/**
* relocate verification logs for volume that's marked down
* ...
*/
void relocateVerificationLogs(FsVolumeSpi volMarkedDown) ...
to make it more clear?
5. In BlockPoolSliceScanner.java,
void relocateVerificationLogs(FsVolumeSpi vol) throws IOException {
if (verificationLog != null) {
// block of code
}
// no code here
}
If the block of code is large, it would be helpful to change
it to
void relocateVerificationLogs(FsVolumeSpi vol) throws IOException {
if (verificationLog == null) {
return;
}
// block of code
}
This helps removing one level of indentation, to make it easier to read.
Thanks.
> Means of telling the datanode to stop using a sick disk
> ---
>
> Key: HDFS-4239
> URL: https://issues.apache.org/jira/browse/HDFS-4239
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: stack
>Assignee: Jimmy Xiang
> Attachments: hdfs-4239.patch, hdfs-4239_v2.patch, hdfs-4239_v3.patch,
> hdfs-4239_v4.patch, hdfs-4239_v5.patch
>
>
> If a disk has been deemed 'sick' -- i.e. not dead but wounded, failing
> occasionally, or just exhibiting high latency -- your choices are:
> 1. Decommission the total datanode. If the datanode is carrying 6 or 12
> disks of data, especially on a cluster that is smallish -- 5 to 20 nodes --
> the rereplication of the downed datanode's data can be pretty disruptive,
> especially if the cluster is doing low latency serving: e.g. hosting an hbase
> cluster.
> 2. Stop the datanode, unmount the bad disk, and restart the datanode (You
> can't unmount the disk while it is in use). This latter is better in that
> only the bad disk's data is rereplicated, not all datanode data.
> Is it possible to do better, say, send the datanode a signal to tell it stop
> using a disk an operator has designated 'bad'. This would be like option #2
> above minus the need to stop and restart the datanode. Ideally the disk
> would become unmountable after a while.
> Nice to have would be being able to tell the datanode to restart using a disk
> after its been replaced.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5954) Merge Protobuf-based-FSImage code from trunk
[ https://issues.apache.org/jira/browse/HDFS-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901851#comment-13901851 ] Jing Zhao commented on HDFS-5954: - I guess the above comment should be posted in HDFS-5698, [~wheat9] ? :) :) > Merge Protobuf-based-FSImage code from trunk > > > Key: HDFS-5954 > URL: https://issues.apache.org/jira/browse/HDFS-5954 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-5954.patch > > > After merging the protobuf-based-fsimage code from trunk, we need to fix some > compilation errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5954) Merge Protobuf-based-FSImage code from trunk
[ https://issues.apache.org/jira/browse/HDFS-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5954: - Attachment: (was: HDFS-5954-branch2.000.patch) > Merge Protobuf-based-FSImage code from trunk > > > Key: HDFS-5954 > URL: https://issues.apache.org/jira/browse/HDFS-5954 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-5954.patch > > > After merging the protobuf-based-fsimage code from trunk, we need to fix some > compilation errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901843#comment-13901843
]
Jing Zhao commented on HDFS-5776:
-
Thanks for updating the patch [~stack]. The latest patch looks great to me.
Just some minors:
# In DFSClient#initThreadNumForHedgedReads, do we need to check whether
HEDGED_READ_THREAD_POOL has already been created?
# nit: we may need an @Override annotation here?
{code}
+new ThreadPoolExecutor.CallerRunsPolicy() {
+ public void rejectedExecution(Runnable runnable,
+ ThreadPoolExecutor e) {
+LOG.info("Execution rejected, Executing in current thread");
+HEDGED_READ_METRIC.incHedgedReadOpsInCurThread();
+// will run in the current thread
+super.rejectedExecution(runnable, e);
+ }
{code}
+1 after addressing the comments.
> Support 'hedged' reads in DFSClient
> ---
>
> Key: HDFS-5776
> URL: https://issues.apache.org/jira/browse/HDFS-5776
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 3.0.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: HDFS-5776-v10.txt, HDFS-5776-v11.txt, HDFS-5776-v12.txt,
> HDFS-5776-v12.txt, HDFS-5776-v13.wip.txt, HDFS-5776-v14.txt,
> HDFS-5776-v15.txt, HDFS-5776-v17.txt, HDFS-5776-v17.txt, HDFS-5776-v2.txt,
> HDFS-5776-v3.txt, HDFS-5776-v4.txt, HDFS-5776-v5.txt, HDFS-5776-v6.txt,
> HDFS-5776-v7.txt, HDFS-5776-v8.txt, HDFS-5776-v9.txt, HDFS-5776.txt
>
>
> This is a placeholder of hdfs related stuff backport from
> https://issues.apache.org/jira/browse/HBASE-7509
> The quorum read ability should be helpful especially to optimize read outliers
> we can utilize "dfs.dfsclient.quorum.read.threshold.millis" &
> "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read
> ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we
> could export the interested metric valus into client system(e.g. HBase's
> regionserver metric).
> The core logic is in pread code path, we decide to goto the original
> fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per
> the above config items.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5698) Use protobuf to serialize / deserialize FSImage
[ https://issues.apache.org/jira/browse/HDFS-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5698: - Attachment: HDFS-5698-branch2.000.patch Merge patch for branch-2. > Use protobuf to serialize / deserialize FSImage > --- > > Key: HDFS-5698 > URL: https://issues.apache.org/jira/browse/HDFS-5698 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0 > > Attachments: HDFS-5698-branch2.000.patch, HDFS-5698-design.pdf, > HDFS-5698.000.patch, HDFS-5698.001.patch, HDFS-5698.002.patch, > HDFS-5698.003.patch, HDFS-5698.004.patch, HDFS-5698.005.patch, > HDFS-5698.006.patch, HDFS-5698.007.patch > > > Currently, the code serializes FSImage using in-house serialization > mechanisms. There are a couple disadvantages of the current approach: > # Mixing the responsibility of reconstruction and serialization / > deserialization. The current code paths of serialization / deserialization > have spent a lot of effort on maintaining compatibility. What is worse is > that they are mixed with the complex logic of reconstructing the namespace, > making the code difficult to follow. > # Poor documentation of the current FSImage format. The format of the FSImage > is practically defined by the implementation. An bug in implementation means > a bug in the specification. Furthermore, it also makes writing third-party > tools quite difficult. > # Changing schemas is non-trivial. Adding a field in FSImage requires bumping > the layout version every time. Bumping out layout version requires (1) the > users to explicitly upgrade the clusters, and (2) putting new code to > maintain backward compatibility. > This jira proposes to use protobuf to serialize the FSImage. Protobuf has > been used to serialize / deserialize the RPC message in Hadoop. > Protobuf addresses all the above problems. It clearly separates the > responsibility of serialization and reconstructing the namespace. The > protobuf files document the current format of the FSImage. The developers now > can add optional fields with ease, since the old code can always read the new > FSImage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5954) Merge Protobuf-based-FSImage code from trunk
[
https://issues.apache.org/jira/browse/HDFS-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-5954:
-
Attachment: HDFS-5954-branch2.000.patch
The v0 patch merges the protobuf code into branch-2. It also includes fixes for
{{TestOfflineImageViewer}}.
> Merge Protobuf-based-FSImage code from trunk
>
>
> Key: HDFS-5954
> URL: https://issues.apache.org/jira/browse/HDFS-5954
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, ha, hdfs-client, namenode
>Reporter: Jing Zhao
>Assignee: Jing Zhao
> Attachments: HDFS-5954.patch
>
>
> After merging the protobuf-based-fsimage code from trunk, we need to fix some
> compilation errors.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5498) Improve datanode startup time
[ https://issues.apache.org/jira/browse/HDFS-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901817#comment-13901817 ] Brandon Li commented on HDFS-5498: -- +1. The patch looks good to me. > Improve datanode startup time > - > > Key: HDFS-5498 > URL: https://issues.apache.org/jira/browse/HDFS-5498 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-5498.with_du_change.patch, > HDFS-5498.with_du_change.patch, HDFS-5498_sh.patch > > > Similarly to HDFS-5027, an improvement can be made for getVomeMap(). This is > the phase in which ReplicaMap.is populated. But it will be even better if > datanode scans only once and do both. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5954) Merge Protobuf-based-FSImage code from trunk
[ https://issues.apache.org/jira/browse/HDFS-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901801#comment-13901801 ] Jing Zhao commented on HDFS-5954: - The patch can fix the compilation error. We still need to fix TestOfflineEditsViewer. But I guess we can also do that in the end in case that we make further changes to editlog. > Merge Protobuf-based-FSImage code from trunk > > > Key: HDFS-5954 > URL: https://issues.apache.org/jira/browse/HDFS-5954 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-5954.patch > > > After merging the protobuf-based-fsimage code from trunk, we need to fix some > compilation errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5954) Merge Protobuf-based-FSImage code from trunk
[ https://issues.apache.org/jira/browse/HDFS-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5954: Attachment: HDFS-5954.patch > Merge Protobuf-based-FSImage code from trunk > > > Key: HDFS-5954 > URL: https://issues.apache.org/jira/browse/HDFS-5954 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-5954.patch > > > After merging the protobuf-based-fsimage code from trunk, we need to fix some > compilation errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5954) Merge Protobuf-based-FSImage code from trunk
Jing Zhao created HDFS-5954: --- Summary: Merge Protobuf-based-FSImage code from trunk Key: HDFS-5954 URL: https://issues.apache.org/jira/browse/HDFS-5954 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Jing Zhao After merging the protobuf-based-fsimage code from trunk, we need to fix some compilation errors. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5775) Consolidate the code for serialization in CacheManager
[ https://issues.apache.org/jira/browse/HDFS-5775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5775: Fix Version/s: 2.4.0 Merged to branch-2. > Consolidate the code for serialization in CacheManager > -- > > Key: HDFS-5775 > URL: https://issues.apache.org/jira/browse/HDFS-5775 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0 >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5775.000.patch > > > This jira proposes to consolidate the code that is responsible for > serializing / deserializing cache manager state into a separate class, so > that it is easier to introduce new code path to serialize the data using > protobuf. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5768) Consolidate the serialization code in DelegationTokenSecretManager
[ https://issues.apache.org/jira/browse/HDFS-5768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5768: Fix Version/s: 2.4.0 Merged to branch-2. > Consolidate the serialization code in DelegationTokenSecretManager > -- > > Key: HDFS-5768 > URL: https://issues.apache.org/jira/browse/HDFS-5768 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0 >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5768.000.patch, HDFS-5768.001.patch > > > This jira proposes to extract a private class for the serialization code for > DelegationTokenSecretManager, so that it becomes easier to introduce new code > paths to serialize the same set of information using protobuf. > This jira does not intend to introduce any functionality changes. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5952) Create a tool to run data analysis on the PB format fsimage
[ https://issues.apache.org/jira/browse/HDFS-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-5952: Description: Delimited processor in OfflineImageViewer is not supported after HDFS-5698 was merged. The motivation of delimited processor is to run data analysis on the fsimage, therefore, there might be more values to create a tool for Hive or Pig that reads the PB format fsimage directly. was: Delimited processor is not supported after HDFS-5698 was merged. The processor is useful for analyzing the output by scripts such as pig. > Create a tool to run data analysis on the PB format fsimage > --- > > Key: HDFS-5952 > URL: https://issues.apache.org/jira/browse/HDFS-5952 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: tools >Affects Versions: 3.0.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > > Delimited processor in OfflineImageViewer is not supported after HDFS-5698 > was merged. > The motivation of delimited processor is to run data analysis on the fsimage, > therefore, there might be more values to create a tool for Hive or Pig that > reads the PB format fsimage directly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5952) Create a tool to run data analysis on the PB format fsimage
[ https://issues.apache.org/jira/browse/HDFS-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-5952: Issue Type: Improvement (was: Sub-task) Parent: (was: HDFS-5863) > Create a tool to run data analysis on the PB format fsimage > --- > > Key: HDFS-5952 > URL: https://issues.apache.org/jira/browse/HDFS-5952 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 3.0.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > > Delimited processor in OfflineImageViewer is not supported after HDFS-5698 > was merged. > The motivation of delimited processor is to run data analysis on the fsimage, > therefore, there might be more values to create a tool for Hive or Pig that > reads the PB format fsimage directly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5952) Create a tool to run data analysis on the PB format fsimage
[ https://issues.apache.org/jira/browse/HDFS-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-5952: Summary: Create a tool to run data analysis on the PB format fsimage (was: Implement delimited processor in OfflineImageViewer) > Create a tool to run data analysis on the PB format fsimage > --- > > Key: HDFS-5952 > URL: https://issues.apache.org/jira/browse/HDFS-5952 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: tools >Affects Versions: 3.0.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > > Delimited processor is not supported after HDFS-5698 was merged. > The processor is useful for analyzing the output by scripts such as pig. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5943) 'dfs.namenode.https-address.ns1' property is not used in federation setup
[ https://issues.apache.org/jira/browse/HDFS-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-5943: -- Resolution: Fixed Fix Version/s: 2.4.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed the patch to trunk and branch-2. Thank you [~jingzhao] for the review. > 'dfs.namenode.https-address.ns1' property is not used in federation setup > - > > Key: HDFS-5943 > URL: https://issues.apache.org/jira/browse/HDFS-5943 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Suresh Srinivas > Fix For: 2.4.0 > > Attachments: HDFS-5943.1.patch, HDFS-5943.patch > > > When federation is ON and two name nodes exist in Cluster, > dfs.namenode.https-address property is not being respected when > dfs.http.policy= HTTPS_ONLY > Scenario: > Pre condition: > Cluster with one namenode is running. The dfs.namenode.https-address is set > to 50701. Namenode UI is accessible at https://NN:50701 > Steps followed: > 1) Enable Federation and start new NN. > 2) Set https addresses for both NNs. > dfs.federation.nameservicesns1,ns2 > dfs.namenode.https-address.ns2host2:50701 > dfs.namenode.https-address.ns1host1:50701 > 3) restart Existing and New namenodes. > Expected behavior: > Both the name nodes are expected to be accessed on 50701 port. Instead both > the Name nodes start with 50470 port. > https://NN:50470 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5943) 'dfs.namenode.https-address.ns1' property is not used in federation setup
[ https://issues.apache.org/jira/browse/HDFS-5943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901727#comment-13901727 ] Hudson commented on HDFS-5943: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5170 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5170/]) HDFS-5943. 'dfs.namenode.https-address' property is not loaded from configuration in federation setup. Contributed by Suresh Srinivas. (suresh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568412) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java > 'dfs.namenode.https-address.ns1' property is not used in federation setup > - > > Key: HDFS-5943 > URL: https://issues.apache.org/jira/browse/HDFS-5943 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yesha Vora >Assignee: Suresh Srinivas > Fix For: 2.4.0 > > Attachments: HDFS-5943.1.patch, HDFS-5943.patch > > > When federation is ON and two name nodes exist in Cluster, > dfs.namenode.https-address property is not being respected when > dfs.http.policy= HTTPS_ONLY > Scenario: > Pre condition: > Cluster with one namenode is running. The dfs.namenode.https-address is set > to 50701. Namenode UI is accessible at https://NN:50701 > Steps followed: > 1) Enable Federation and start new NN. > 2) Set https addresses for both NNs. > dfs.federation.nameservicesns1,ns2 > dfs.namenode.https-address.ns2host2:50701 > dfs.namenode.https-address.ns1host1:50701 > 3) restart Existing and New namenodes. > Expected behavior: > Both the name nodes are expected to be accessed on 50701 port. Instead both > the Name nodes start with 50470 port. > https://NN:50470 -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5585) Provide admin commands for data node upgrade
[ https://issues.apache.org/jira/browse/HDFS-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901722#comment-13901722 ] Brandon Li commented on HDFS-5585: -- +1 > Provide admin commands for data node upgrade > > > Key: HDFS-5585 > URL: https://issues.apache.org/jira/browse/HDFS-5585 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, ha, hdfs-client, namenode >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-5585.patch, HDFS-5585.patch, HDFS-5585.patch > > > Several new methods to ClientDatanodeProtocol may need to be added to support > querying version, initiating upgrade, etc. The admin CLI needs to be added > as well. This primary use case is for rolling upgrade, but this can be used > for preparing for a graceful restart of a data node for any reasons. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5944) LeaseManager:findLeaseWithPrefixPath didn't handle path like /a/b/ right cause SecondaryNameNode failed do checkpoint
[
https://issues.apache.org/jira/browse/HDFS-5944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901718#comment-13901718
]
Brandon Li commented on HDFS-5944:
--
{quote}1. Is it enough for just writing a unit test for
findLeaseWithPrefixPath?{quote}
Please feel free to include the unit test uploaded yesterday. You can also add
more test steps, such as using FileSystem object to delete path (e.g.,
"/a/b/../.") as you mentioned.
{quote}2. In trunk, there is no TestLeaseManager.java, should I add one?{quote}
You can add the unit test to TestLease.java.
> LeaseManager:findLeaseWithPrefixPath didn't handle path like /a/b/ right
> cause SecondaryNameNode failed do checkpoint
> -
>
> Key: HDFS-5944
> URL: https://issues.apache.org/jira/browse/HDFS-5944
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 1.2.0, 2.2.0
>Reporter: zhaoyunjiong
>Assignee: zhaoyunjiong
> Attachments: HDFS-5944-branch-1.2.patch, HDFS-5944.patch,
> HDFS-5944.test.txt
>
>
> In our cluster, we encountered error like this:
> java.io.IOException: saveLeases found path
> /XXX/20140206/04_30/_SUCCESS.slc.log but is not under construction.
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:6217)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:607)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveCurrent(FSImage.java:1004)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveNamespace(FSImage.java:949)
> What happened:
> Client A open file /XXX/20140206/04_30/_SUCCESS.slc.log for write.
> And Client A continue refresh it's lease.
> Client B deleted /XXX/20140206/04_30/
> Client C open file /XXX/20140206/04_30/_SUCCESS.slc.log for write
> Client C closed the file /XXX/20140206/04_30/_SUCCESS.slc.log
> Then secondaryNameNode try to do checkpoint and failed due to failed to
> delete lease hold by Client A when Client B deleted /XXX/20140206/04_30/.
> The reason is a bug in findLeaseWithPrefixPath:
> int srclen = prefix.length();
> if (p.length() == srclen || p.charAt(srclen) == Path.SEPARATOR_CHAR) {
> entries.put(entry.getKey(), entry.getValue());
> }
> Here when prefix is /XXX/20140206/04_30/, and p is
> /XXX/20140206/04_30/_SUCCESS.slc.log, p.charAt(srcllen) is '_'.
> The fix is simple, I'll upload patch later.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
