[jira] [Commented] (HDFS-5362) Add SnapshotException to terse exception group
[
https://issues.apache.org/jira/browse/HDFS-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798853#comment-13798853
]
Hadoop QA commented on HDFS-5362:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609093/HDFS-5362.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5232//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5232//console
This message is automatically generated.
> Add SnapshotException to terse exception group
> --
>
> Key: HDFS-5362
> URL: https://issues.apache.org/jira/browse/HDFS-5362
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 3.0.0
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HDFS-5362.patch
>
>
> In trunk, a stack trace of SnapshotException is output NameNode's log via
> ipc.Server class.
> The trace of the output method is easy for the message of SnapshotException.
> So, it should add SnapshotException to terse exception group of
> NameNodeRpcServer.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5167) Add metrics about the NameNode retry cache
[
https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798901#comment-13798901
]
Hadoop QA commented on HDFS-5167:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609098/HDFS-5167.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5233//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5233//console
This message is automatically generated.
> Add metrics about the NameNode retry cache
> --
>
> Key: HDFS-5167
> URL: https://issues.apache.org/jira/browse/HDFS-5167
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, namenode
>Affects Versions: 3.0.0
>Reporter: Jing Zhao
>Assignee: Tsuyoshi OZAWA
>Priority: Minor
> Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch,
> HDFS-5167.4.patch, HDFS-5167.5.patch, HDFS-5167.6.patch
>
>
> It will be helpful to have metrics in NameNode about the retry cache, such as
> the retry count etc.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-4754) Add an API in the namenode to mark a datanode as stale
[ https://issues.apache.org/jira/browse/HDFS-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-4754: - Fix Version/s: (was: 2.2.0) 2.2.1 > Add an API in the namenode to mark a datanode as stale > -- > > Key: HDFS-4754 > URL: https://issues.apache.org/jira/browse/HDFS-4754 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Critical > Fix For: 3.0.0, 2.2.1 > > Attachments: 4754.v1.patch, 4754.v2.patch, 4754.v4.patch, > 4754.v4.patch > > > There is a detection of the stale datanodes in HDFS since HDFS-3703, with a > timeout, defaulted to 30s. > There are two reasons to add an API to mark a node as stale even if the > timeout is not yet reached: > 1) ZooKeeper can detect that a client is dead at any moment. So, for HBase, > we sometimes start the recovery before a node is marked staled. (even with > reasonable settings as: stale: 20s; HBase ZK timeout: 30s > 2) Some third parties could detect that a node is dead before the timeout, > hence saving us the cost of retrying. An example or such hw is Arista, > presented here by [~tsuna] > http://tsunanet.net/~tsuna/fsf-hbase-meetup-april13.pdf, and confirmed in > HBASE-6290. > As usual, even if the node is dead it can comeback before the 10 minutes > limit. So I would propose to set a timebound. The API would be > namenode.markStale(String ipAddress, int port, long durationInMs); > After durationInMs, the namenode would again rely only on its heartbeat to > decide. > Thoughts? > If there is no objections, and if nobody in the hdfs dev team has the time to > spend some time on it, I will give it a try for branch 2 & 3. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5244) TestNNStorageRetentionManager#testPurgeMultipleDirs fails because incorrectly expects Hashmap values to have order.
[ https://issues.apache.org/jira/browse/HDFS-5244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-5244: - Fix Version/s: (was: 2.2.0) 2.2.1 > TestNNStorageRetentionManager#testPurgeMultipleDirs fails because incorrectly > expects Hashmap values to have order. > > > Key: HDFS-5244 > URL: https://issues.apache.org/jira/browse/HDFS-5244 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.0-beta, 2.2.1 > > Attachments: HDFS-5244.patch > > > The test o.a.h.hdfs.server.namenode.TestNNStorageRetentionManager uses a > HashMap(dirRoots) to store the root storages to be mocked for the purging > test, which does not have any predictable order. The directories needs be > purged are stored in a LinkedHashSet, which has a predictable order. So, when > the directories get mocked for the test, they could be already out of > the order that they were added. Thus, the order that the directories were > actually purged and the order of them being added to the LinkedHashList could > be different and cause the test to fail. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5098) Enhance FileSystem.Statistics to have locality information
[ https://issues.apache.org/jira/browse/HDFS-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HDFS-5098: - Fix Version/s: (was: 2.2.0) 2.2.1 > Enhance FileSystem.Statistics to have locality information > -- > > Key: HDFS-5098 > URL: https://issues.apache.org/jira/browse/HDFS-5098 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Suresh Srinivas > Fix For: 2.2.1 > > > Currently in MR/Tez we dont have a good and accurate means to detect how much > the the IO was actually done locally. Getting this information from the > source of truth would be much better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5160) MiniDFSCluster webui does not work
[
https://issues.apache.org/jira/browse/HDFS-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-5160:
-
Fix Version/s: (was: 2.2.0)
2.2.1
> MiniDFSCluster webui does not work
> --
>
> Key: HDFS-5160
> URL: https://issues.apache.org/jira/browse/HDFS-5160
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.1.0-beta
>Reporter: Alejandro Abdelnur
> Fix For: 2.2.1
>
>
> The webui does not work, when going to http://localhost:50070 you get:
> {code}
> Directory: /
> webapps/ 102 bytes Sep 4, 2013 9:32:55 AM
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5167) Add metrics about the NameNode retry cache
[ https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HDFS-5167: - Attachment: HDFS-5167.6.patch TestMetricsSystemImpl's test failure looks unrelated to me. Test again with the same patch. > Add metrics about the NameNode retry cache > -- > > Key: HDFS-5167 > URL: https://issues.apache.org/jira/browse/HDFS-5167 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, namenode >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch, > HDFS-5167.4.patch, HDFS-5167.5.patch, HDFS-5167.6.patch, HDFS-5167.6.patch > > > It will be helpful to have metrics in NameNode about the retry cache, such as > the retry count etc. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5167) Add metrics about the NameNode retry cache
[ https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HDFS-5167: - Attachment: HDFS-5167.7.patch I attached wrong patches for last 2 times. I apologize for this. > Add metrics about the NameNode retry cache > -- > > Key: HDFS-5167 > URL: https://issues.apache.org/jira/browse/HDFS-5167 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, namenode >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch, > HDFS-5167.4.patch, HDFS-5167.5.patch, HDFS-5167.6.patch, HDFS-5167.6.patch, > HDFS-5167.7.patch > > > It will be helpful to have metrics in NameNode about the retry cache, such as > the retry count etc. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5167) Add metrics about the NameNode retry cache
[ https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798933#comment-13798933 ] Tsuyoshi OZAWA commented on HDFS-5167: -- HDFS-5167.7.patch is correct one. > Add metrics about the NameNode retry cache > -- > > Key: HDFS-5167 > URL: https://issues.apache.org/jira/browse/HDFS-5167 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, namenode >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch, > HDFS-5167.4.patch, HDFS-5167.5.patch, HDFS-5167.6.patch, HDFS-5167.6.patch, > HDFS-5167.7.patch > > > It will be helpful to have metrics in NameNode about the retry cache, such as > the retry count etc. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5365) Fix libhdfs compile error on FreeBSD9
[ https://issues.apache.org/jira/browse/HDFS-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13798935#comment-13798935 ] Radim Kolar commented on HDFS-5365: --- can be this merged to branch-2.1-beta ? > Fix libhdfs compile error on FreeBSD9 > - > > Key: HDFS-5365 > URL: https://issues.apache.org/jira/browse/HDFS-5365 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Radim Kolar >Assignee: Radim Kolar > Labels: build > Fix For: 3.0.0, 2.3.0 > > Attachments: hdfs-bsd.txt > > > native library do not compiles on freebsd because: > * dlopen is in libc > * limits.h include is missing -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5341) DirectoryScanner hold the object lock dataset too long in scan,it make the datanode turn into deadnode and block all reciving thread
[
https://issues.apache.org/jira/browse/HDFS-5341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799001#comment-13799001
]
qus-jiawei commented on HDFS-5341:
--
Hi Prakash :Thanks for help.
In out Hadoop cluster, some datanode store nearly 800 thousand blocks and 16
Disks.
when the disk's IO with high presure,the file getlength funciotn could be slow.
And it will hold the lock dataset too long.
> DirectoryScanner hold the object lock dataset too long in scan,it make the
> datanode turn into deadnode and block all reciving thread
> -
>
> Key: HDFS-5341
> URL: https://issues.apache.org/jira/browse/HDFS-5341
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.0.4-alpha
>Reporter: qus-jiawei
>
> When DirectoryScanner doing the scan function, it holding the dataset to diff
> the block info between memory and disk.But it do a lot of disk operation
> because it call the file's getlength funciton.
> So,such disk operation should move to the diskreport.
> I don't have an commit acount.So I post the patch here.
> 163d162
> < private final long blockFileLength;
> 174,179d172
> < if( this.blockFile != null ){
> < this.blockFileLength = this.blockFile.length();
> < }
> < else{
> < this.blockFileLength = 0;
> < }
> 190,193d182
> <
> < long getBlockFileLength(){
> < return blockFileLength;
> < }
> 249,256c238
> < new Daemon.DaemonFactory(){
> < @Override
> < public Thread newThread(Runnable
> runnable) {
> < Thread
> t=super.newThread(runnable);
> <
> t.setPriority(Thread.NORM_PRIORITY);
> < return t;
> < }
> < });
> ---
> > new Daemon.DaemonFactory());
> 358d339
> < LOG.info("UCADD check and update finish");
> 368d348
> < long begin = System.currentTimeMillis();
> 370c350
> < LOG.info("UCADD finish diskReport
> using:"+(System.currentTimeMillis()-begin)+"ms");
> ---
> >
> 373,375d352
> < begin = System.currentTimeMillis();
> < int diskHit = 0;
> < LOG.info("UCADD begin to synchronized");
> 415,423c392,396
> < || info.getBlockFileLength() !=
> memBlock.getNumBytes() ) {
> < //double check the block file length
> < diskHit++;
> < if(info.getBlockFile().length() !=
> memBlock.getNumBytes()){
> < // Block metadata file is missing or has
> wrong generation stamp,
> < // or block file length is different than
> expected
> < statsRecord.mismatchBlocks++;
> < addDifference(diffRecord, statsRecord,
> info);
> < }
> ---
> > || info.getBlockFile().length() !=
> memBlock.getNumBytes()) {
> > // Block metadata file is missing or has wrong
> generation stamp,
> > // or block file length is different than expected
> > statsRecord.mismatchBlocks++;
> > addDifference(diffRecord, statsRecord, info);
> 437d409
> < LOG.info("UCADD end synchronized
> using:"+(System.currentTimeMillis()-begin)+"ms diskHit:"+diskHit);
> 439d410
> <
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4511) Cover package org.apache.hadoop.hdfs.tools with unit test
[ https://issues.apache.org/jira/browse/HDFS-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799006#comment-13799006 ] Hudson commented on HDFS-4511: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #366 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/366/]) HDFS-4511. Cover package org.apache.hadoop.hdfs.tools with unit test (Andrey Klochkov via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533270) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/JMXGet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenRemoteFetcher.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestJMXGet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestTools.java > Cover package org.apache.hadoop.hdfs.tools with unit test > - > > Key: HDFS-4511 > URL: https://issues.apache.org/jira/browse/HDFS-4511 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Vadim Bondarev >Assignee: Andrey Klochkov > Attachments: HADOOP-4511-branch-0.23-a.patch, > HADOOP-4511-branch-2-a.patch, HADOOP-4511-trunk-a.patch, > HDFS-4511-branch-2--N2.patch, HDFS-4511--n6.patch, HDFS-4511-trunk--N4.patch, > HDFS-4511-trunk--n5.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5365) Fix libhdfs compile error on FreeBSD9
[ https://issues.apache.org/jira/browse/HDFS-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799009#comment-13799009 ] Hudson commented on HDFS-5365: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #366 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/366/]) HDFS-5365. Fix libhdfs compile error on FreeBSD9. Contributed by Radim Kolar. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533283) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/posix_util.c > Fix libhdfs compile error on FreeBSD9 > - > > Key: HDFS-5365 > URL: https://issues.apache.org/jira/browse/HDFS-5365 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Radim Kolar >Assignee: Radim Kolar > Labels: build > Fix For: 3.0.0, 2.3.0 > > Attachments: hdfs-bsd.txt > > > native library do not compiles on freebsd because: > * dlopen is in libc > * limits.h include is missing -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5379) Update links to datanode information in dfshealth.html
[ https://issues.apache.org/jira/browse/HDFS-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799005#comment-13799005 ] Hudson commented on HDFS-5379: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #366 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/366/]) HDFS-5379. Update links to datanode information in dfshealth.html. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533181) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.dust.html > Update links to datanode information in dfshealth.html > -- > > Key: HDFS-5379 > URL: https://issues.apache.org/jira/browse/HDFS-5379 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.3.0 > > Attachments: HDFS-5379.000.patch > > > Currently dfshealth.html already provides the information listed in > dfsnodelist.jsp. The links to datanode information in dfshealth.html should > be updated. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5336) DataNode should not output 'StartupProgress' metrics
[
https://issues.apache.org/jira/browse/HDFS-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799007#comment-13799007
]
Hudson commented on HDFS-5336:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #366 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/366/])
HDFS-5336. DataNode should not output StartupProgress metrics. Contributed by
Akira Ajisaka. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533183)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> DataNode should not output 'StartupProgress' metrics
>
>
> Key: HDFS-5336
> URL: https://issues.apache.org/jira/browse/HDFS-5336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.1.0-beta
> Environment: trunk
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: metrics
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HDFS-5336.2.patch, HDFS-5336.patch
>
>
> I found the following metrics output from DataNode.
> {code}
> 1381355455731 default.StartupProgress: Hostname=trunk, ElapsedTime=0,
> PercentComplete=0.0, LoadingFsImageCount=0, LoadingFsImageElapsedTime=0,
> LoadingFsImageTotal=0, LoadingFsImagePercentComplete=0.0,
> LoadingEditsCount=0, LoadingEditsElapsedTime=0, LoadingEditsTotal=0,
> LoadingEditsPercentComplete=0.0, SavingCheckpointCount=0,
> SavingCheckpointElapsedTime=0, SavingCheckpointTotal=0,
> SavingCheckpointPercentComplete=0.0, SafeModeCount=0, SafeModeElapsedTime=0,
> SafeModeTotal=0, SafeModePercentComplete=0.0
> {code}
> DataNode should not output 'StartupProgress' metrics because the metrics
> shows the progress of NameNode startup.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5374) Remove deadcode in DFSOutputStream
[
https://issues.apache.org/jira/browse/HDFS-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799004#comment-13799004
]
Hudson commented on HDFS-5374:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #366 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/366/])
HDFS-5374. Remove deadcode in DFSOutputStream. Contributed by Suresh Srinivas.
(suresh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533258)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> Remove deadcode in DFSOutputStream
> --
>
> Key: HDFS-5374
> URL: https://issues.apache.org/jira/browse/HDFS-5374
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
>Priority: Trivial
> Fix For: 2.3.0
>
> Attachments: HDFS-4374.patch
>
>
> Deadcode:
> {code}
> if (one.isHeartbeatPacket()) { //heartbeat packet
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5375) hdfs.cmd does not expose several snapshot commands.
[ https://issues.apache.org/jira/browse/HDFS-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799003#comment-13799003 ] Hudson commented on HDFS-5375: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #366 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/366/]) HDFS-5375. hdfs.cmd does not expose several snapshot commands. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533165) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd > hdfs.cmd does not expose several snapshot commands. > --- > > Key: HDFS-5375 > URL: https://issues.apache.org/jira/browse/HDFS-5375 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.2.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HDFS-5375.1.patch > > > We need to update hdfs.cmd to expose the snapshotDiff and lsSnapshottableDir > commands on Windows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5239) Allow FSNamesystem lock fairness to be configurable
[
https://issues.apache.org/jira/browse/HDFS-5239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799029#comment-13799029
]
Hudson commented on HDFS-5239:
--
FAILURE: Integrated in Hadoop-Hdfs-0.23-Build #764 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/764/])
HDFS-5239. Allow FSNamesystem lock fairness to be configurable. Contributed by
Daryn Sharp. (kihwal:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533252)
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
> Allow FSNamesystem lock fairness to be configurable
> ---
>
> Key: HDFS-5239
> URL: https://issues.apache.org/jira/browse/HDFS-5239
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 3.0.0, 2.3.0, 0.23.10
>
> Attachments: HADOOP-5239.branach-23.patch, HDFS-5239.patch
>
>
> The fairness of the {{FSNamesystem#fsLock}} is hardcoded to {{true}}. Using
> an unfair lock provides a negligible increase to throughput. However this is
> due to bottlenecks elsewhere in the system. In combination with other
> changes, such as RPC layer and audit logging, preliminary tests show up to a
> 5X improvement for a read heavy workload.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4511) Cover package org.apache.hadoop.hdfs.tools with unit test
[ https://issues.apache.org/jira/browse/HDFS-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799033#comment-13799033 ] Hudson commented on HDFS-4511: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1556 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1556/]) HDFS-4511. Cover package org.apache.hadoop.hdfs.tools with unit test (Andrey Klochkov via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533270) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/JMXGet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenRemoteFetcher.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestJMXGet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestTools.java > Cover package org.apache.hadoop.hdfs.tools with unit test > - > > Key: HDFS-4511 > URL: https://issues.apache.org/jira/browse/HDFS-4511 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Vadim Bondarev >Assignee: Andrey Klochkov > Attachments: HADOOP-4511-branch-0.23-a.patch, > HADOOP-4511-branch-2-a.patch, HADOOP-4511-trunk-a.patch, > HDFS-4511-branch-2--N2.patch, HDFS-4511--n6.patch, HDFS-4511-trunk--N4.patch, > HDFS-4511-trunk--n5.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5379) Update links to datanode information in dfshealth.html
[ https://issues.apache.org/jira/browse/HDFS-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799032#comment-13799032 ] Hudson commented on HDFS-5379: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1556 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1556/]) HDFS-5379. Update links to datanode information in dfshealth.html. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533181) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.dust.html > Update links to datanode information in dfshealth.html > -- > > Key: HDFS-5379 > URL: https://issues.apache.org/jira/browse/HDFS-5379 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.3.0 > > Attachments: HDFS-5379.000.patch > > > Currently dfshealth.html already provides the information listed in > dfsnodelist.jsp. The links to datanode information in dfshealth.html should > be updated. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5374) Remove deadcode in DFSOutputStream
[
https://issues.apache.org/jira/browse/HDFS-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799031#comment-13799031
]
Hudson commented on HDFS-5374:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1556 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1556/])
HDFS-5374. Remove deadcode in DFSOutputStream. Contributed by Suresh Srinivas.
(suresh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533258)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> Remove deadcode in DFSOutputStream
> --
>
> Key: HDFS-5374
> URL: https://issues.apache.org/jira/browse/HDFS-5374
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
>Priority: Trivial
> Fix For: 2.3.0
>
> Attachments: HDFS-4374.patch
>
>
> Deadcode:
> {code}
> if (one.isHeartbeatPacket()) { //heartbeat packet
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5365) Fix libhdfs compile error on FreeBSD9
[ https://issues.apache.org/jira/browse/HDFS-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799036#comment-13799036 ] Hudson commented on HDFS-5365: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1556 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1556/]) HDFS-5365. Fix libhdfs compile error on FreeBSD9. Contributed by Radim Kolar. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533283) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/posix_util.c > Fix libhdfs compile error on FreeBSD9 > - > > Key: HDFS-5365 > URL: https://issues.apache.org/jira/browse/HDFS-5365 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Radim Kolar >Assignee: Radim Kolar > Labels: build > Fix For: 3.0.0, 2.3.0 > > Attachments: hdfs-bsd.txt > > > native library do not compiles on freebsd because: > * dlopen is in libc > * limits.h include is missing -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5375) hdfs.cmd does not expose several snapshot commands.
[ https://issues.apache.org/jira/browse/HDFS-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799030#comment-13799030 ] Hudson commented on HDFS-5375: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1556 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1556/]) HDFS-5375. hdfs.cmd does not expose several snapshot commands. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533165) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd > hdfs.cmd does not expose several snapshot commands. > --- > > Key: HDFS-5375 > URL: https://issues.apache.org/jira/browse/HDFS-5375 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.2.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HDFS-5375.1.patch > > > We need to update hdfs.cmd to expose the snapshotDiff and lsSnapshottableDir > commands on Windows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5336) DataNode should not output 'StartupProgress' metrics
[
https://issues.apache.org/jira/browse/HDFS-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799034#comment-13799034
]
Hudson commented on HDFS-5336:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1556 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1556/])
HDFS-5336. DataNode should not output StartupProgress metrics. Contributed by
Akira Ajisaka. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533183)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> DataNode should not output 'StartupProgress' metrics
>
>
> Key: HDFS-5336
> URL: https://issues.apache.org/jira/browse/HDFS-5336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.1.0-beta
> Environment: trunk
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: metrics
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HDFS-5336.2.patch, HDFS-5336.patch
>
>
> I found the following metrics output from DataNode.
> {code}
> 1381355455731 default.StartupProgress: Hostname=trunk, ElapsedTime=0,
> PercentComplete=0.0, LoadingFsImageCount=0, LoadingFsImageElapsedTime=0,
> LoadingFsImageTotal=0, LoadingFsImagePercentComplete=0.0,
> LoadingEditsCount=0, LoadingEditsElapsedTime=0, LoadingEditsTotal=0,
> LoadingEditsPercentComplete=0.0, SavingCheckpointCount=0,
> SavingCheckpointElapsedTime=0, SavingCheckpointTotal=0,
> SavingCheckpointPercentComplete=0.0, SafeModeCount=0, SafeModeElapsedTime=0,
> SafeModeTotal=0, SafeModePercentComplete=0.0
> {code}
> DataNode should not output 'StartupProgress' metrics because the metrics
> shows the progress of NameNode startup.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5167) Add metrics about the NameNode retry cache
[
https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799038#comment-13799038
]
Hadoop QA commented on HDFS-5167:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609116/HDFS-5167.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5234//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5234//console
This message is automatically generated.
> Add metrics about the NameNode retry cache
> --
>
> Key: HDFS-5167
> URL: https://issues.apache.org/jira/browse/HDFS-5167
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, namenode
>Affects Versions: 3.0.0
>Reporter: Jing Zhao
>Assignee: Tsuyoshi OZAWA
>Priority: Minor
> Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch,
> HDFS-5167.4.patch, HDFS-5167.5.patch, HDFS-5167.6.patch, HDFS-5167.6.patch,
> HDFS-5167.7.patch
>
>
> It will be helpful to have metrics in NameNode about the retry cache, such as
> the retry count etc.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-4311) repair test org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos
[ https://issues.apache.org/jira/browse/HDFS-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HDFS-4311: - Attachment: HDFS-4311--N1.patch Test rewritten to use MiniKdc. > repair test org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos > --- > > Key: HDFS-4311 > URL: https://issues.apache.org/jira/browse/HDFS-4311 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.0.3-alpha >Reporter: Ivan A. Veselovsky >Assignee: Ivan A. Veselovsky > Attachments: HDFS-4311--N1.patch, HDFS-4311.patch > > > Some of the test cases in this test class are failing because they are > affected by static state changed by the previous test cases. Namely this is > the static field org.apache.hadoop.security.UserGroupInformation.loginUser . > The suggested patch solves this problem. > Besides, the following improvements are done: > 1) parametrized the user principal and keytab values via system properties; > 2) shutdown of the Jetty server and the minicluster between the test cases is > added to make the test methods independent on each other. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4311) repair test org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos
[
https://issues.apache.org/jira/browse/HDFS-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799083#comment-13799083
]
Hadoop QA commented on HDFS-4311:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609128/HDFS-4311--N1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs-httpfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5235//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5235//console
This message is automatically generated.
> repair test org.apache.hadoop.fs.http.server.TestHttpFSWithKerberos
> ---
>
> Key: HDFS-4311
> URL: https://issues.apache.org/jira/browse/HDFS-4311
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.0.3-alpha
>Reporter: Ivan A. Veselovsky
>Assignee: Ivan A. Veselovsky
> Attachments: HDFS-4311--N1.patch, HDFS-4311.patch
>
>
> Some of the test cases in this test class are failing because they are
> affected by static state changed by the previous test cases. Namely this is
> the static field org.apache.hadoop.security.UserGroupInformation.loginUser .
> The suggested patch solves this problem.
> Besides, the following improvements are done:
> 1) parametrized the user principal and keytab values via system properties;
> 2) shutdown of the Jetty server and the minicluster between the test cases is
> added to make the test methods independent on each other.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5365) Fix libhdfs compile error on FreeBSD9
[ https://issues.apache.org/jira/browse/HDFS-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799095#comment-13799095 ] Hudson commented on HDFS-5365: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1582 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1582/]) HDFS-5365. Fix libhdfs compile error on FreeBSD9. Contributed by Radim Kolar. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533283) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/native/util/posix_util.c > Fix libhdfs compile error on FreeBSD9 > - > > Key: HDFS-5365 > URL: https://issues.apache.org/jira/browse/HDFS-5365 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Radim Kolar >Assignee: Radim Kolar > Labels: build > Fix For: 3.0.0, 2.3.0 > > Attachments: hdfs-bsd.txt > > > native library do not compiles on freebsd because: > * dlopen is in libc > * limits.h include is missing -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5375) hdfs.cmd does not expose several snapshot commands.
[ https://issues.apache.org/jira/browse/HDFS-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799089#comment-13799089 ] Hudson commented on HDFS-5375: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1582 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1582/]) HDFS-5375. hdfs.cmd does not expose several snapshot commands. Contributed by Chris Nauroth. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533165) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd > hdfs.cmd does not expose several snapshot commands. > --- > > Key: HDFS-5375 > URL: https://issues.apache.org/jira/browse/HDFS-5375 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.2.0 >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Minor > Fix For: 3.0.0, 2.3.0 > > Attachments: HDFS-5375.1.patch > > > We need to update hdfs.cmd to expose the snapshotDiff and lsSnapshottableDir > commands on Windows. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5379) Update links to datanode information in dfshealth.html
[ https://issues.apache.org/jira/browse/HDFS-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799091#comment-13799091 ] Hudson commented on HDFS-5379: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1582 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1582/]) HDFS-5379. Update links to datanode information in dfshealth.html. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533181) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.dust.html > Update links to datanode information in dfshealth.html > -- > > Key: HDFS-5379 > URL: https://issues.apache.org/jira/browse/HDFS-5379 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.3.0 > > Attachments: HDFS-5379.000.patch > > > Currently dfshealth.html already provides the information listed in > dfsnodelist.jsp. The links to datanode information in dfshealth.html should > be updated. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5374) Remove deadcode in DFSOutputStream
[
https://issues.apache.org/jira/browse/HDFS-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799090#comment-13799090
]
Hudson commented on HDFS-5374:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1582 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1582/])
HDFS-5374. Remove deadcode in DFSOutputStream. Contributed by Suresh Srinivas.
(suresh:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533258)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
> Remove deadcode in DFSOutputStream
> --
>
> Key: HDFS-5374
> URL: https://issues.apache.org/jira/browse/HDFS-5374
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
>Priority: Trivial
> Fix For: 2.3.0
>
> Attachments: HDFS-4374.patch
>
>
> Deadcode:
> {code}
> if (one.isHeartbeatPacket()) { //heartbeat packet
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4511) Cover package org.apache.hadoop.hdfs.tools with unit test
[ https://issues.apache.org/jira/browse/HDFS-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799092#comment-13799092 ] Hudson commented on HDFS-4511: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1582 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1582/]) HDFS-4511. Cover package org.apache.hadoop.hdfs.tools with unit test (Andrey Klochkov via jeagles) (jeagles: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533270) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/JMXGet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenRemoteFetcher.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestJMXGet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestTools.java > Cover package org.apache.hadoop.hdfs.tools with unit test > - > > Key: HDFS-4511 > URL: https://issues.apache.org/jira/browse/HDFS-4511 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Vadim Bondarev >Assignee: Andrey Klochkov > Attachments: HADOOP-4511-branch-0.23-a.patch, > HADOOP-4511-branch-2-a.patch, HADOOP-4511-trunk-a.patch, > HDFS-4511-branch-2--N2.patch, HDFS-4511--n6.patch, HDFS-4511-trunk--N4.patch, > HDFS-4511-trunk--n5.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5336) DataNode should not output 'StartupProgress' metrics
[
https://issues.apache.org/jira/browse/HDFS-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799093#comment-13799093
]
Hudson commented on HDFS-5336:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1582 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1582/])
HDFS-5336. DataNode should not output StartupProgress metrics. Contributed by
Akira Ajisaka. (cnauroth:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533183)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
> DataNode should not output 'StartupProgress' metrics
>
>
> Key: HDFS-5336
> URL: https://issues.apache.org/jira/browse/HDFS-5336
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.1.0-beta
> Environment: trunk
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: metrics
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HDFS-5336.2.patch, HDFS-5336.patch
>
>
> I found the following metrics output from DataNode.
> {code}
> 1381355455731 default.StartupProgress: Hostname=trunk, ElapsedTime=0,
> PercentComplete=0.0, LoadingFsImageCount=0, LoadingFsImageElapsedTime=0,
> LoadingFsImageTotal=0, LoadingFsImagePercentComplete=0.0,
> LoadingEditsCount=0, LoadingEditsElapsedTime=0, LoadingEditsTotal=0,
> LoadingEditsPercentComplete=0.0, SavingCheckpointCount=0,
> SavingCheckpointElapsedTime=0, SavingCheckpointTotal=0,
> SavingCheckpointPercentComplete=0.0, SafeModeCount=0, SafeModeElapsedTime=0,
> SafeModeTotal=0, SafeModePercentComplete=0.0
> {code}
> DataNode should not output 'StartupProgress' metrics because the metrics
> shows the progress of NameNode startup.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5386) Add feature documentation for datanode caching.
Chris Nauroth created HDFS-5386: --- Summary: Add feature documentation for datanode caching. Key: HDFS-5386 URL: https://issues.apache.org/jira/browse/HDFS-5386 Project: Hadoop HDFS Issue Type: Improvement Components: documentation Affects Versions: HDFS-4949 Reporter: Chris Nauroth Write feature documentation for datanode caching, covering all of the following: * high-level architecture * OS/native code requirements * OS configuration (ulimit -l) * new configuration properties for namenode and datanode * cache admin CLI commands * pointers to API for programmatic control of caching directives -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Reopened] (HDFS-5365) Fix libhdfs compile error on FreeBSD9
[ https://issues.apache.org/jira/browse/HDFS-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth reopened HDFS-5365: - bq. can be this merged to branch-2.1-beta ? branch-2.1-beta is closed. Now that release 2.2.0 is GA, there won't be any more releases from branch-2.1-beta. However, we can merge to branch-2.2 for inclusion in the 2.2.1 release. I'll do that later today. > Fix libhdfs compile error on FreeBSD9 > - > > Key: HDFS-5365 > URL: https://issues.apache.org/jira/browse/HDFS-5365 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Radim Kolar >Assignee: Radim Kolar > Labels: build > Fix For: 3.0.0, 2.3.0 > > Attachments: hdfs-bsd.txt > > > native library do not compiles on freebsd because: > * dlopen is in libc > * limits.h include is missing -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5365) Fix libhdfs compile error on FreeBSD9
[ https://issues.apache.org/jira/browse/HDFS-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5365: Target Version/s: 3.0.0, 2.2.1 (was: 3.0.0, 2.3.0) > Fix libhdfs compile error on FreeBSD9 > - > > Key: HDFS-5365 > URL: https://issues.apache.org/jira/browse/HDFS-5365 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Radim Kolar >Assignee: Radim Kolar > Labels: build > Fix For: 3.0.0, 2.3.0 > > Attachments: hdfs-bsd.txt > > > native library do not compiles on freebsd because: > * dlopen is in libc > * limits.h include is missing -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4511) Cover package org.apache.hadoop.hdfs.tools with unit test
[ https://issues.apache.org/jira/browse/HDFS-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated HDFS-4511: -- Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Status: Resolved (was: Patch Available) I put this into branch-2 and trunk. Thanks everybody. > Cover package org.apache.hadoop.hdfs.tools with unit test > - > > Key: HDFS-4511 > URL: https://issues.apache.org/jira/browse/HDFS-4511 > Project: Hadoop HDFS > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Vadim Bondarev >Assignee: Andrey Klochkov > Fix For: 3.0.0, 2.3.0 > > Attachments: HADOOP-4511-branch-0.23-a.patch, > HADOOP-4511-branch-2-a.patch, HADOOP-4511-trunk-a.patch, > HDFS-4511-branch-2--N2.patch, HDFS-4511--n6.patch, HDFS-4511-trunk--N4.patch, > HDFS-4511-trunk--n5.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5387) Documentation on FileSystem.createNewFile() is misleading
Leif Wickland created HDFS-5387: --- Summary: Documentation on FileSystem.createNewFile() is misleading Key: HDFS-5387 URL: https://issues.apache.org/jira/browse/HDFS-5387 Project: Hadoop HDFS Issue Type: Improvement Reporter: Leif Wickland Priority: Trivial Attachments: FileSystem.java.patch The documentation for FileSystem.createNewFile() indicates that it will return false if it failed to create the requested file. In fact, it will only return false if the file existed before the function was called. Also the function fails to mention that a race condition can result in an exception be thrown if another process attempts to create the file at the same time. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5387) Documentation on FileSystem.createNewFile() is misleading
[ https://issues.apache.org/jira/browse/HDFS-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leif Wickland updated HDFS-5387: Attachment: FileSystem.java.patch Here's my suggested change to the document's wording. > Documentation on FileSystem.createNewFile() is misleading > - > > Key: HDFS-5387 > URL: https://issues.apache.org/jira/browse/HDFS-5387 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Leif Wickland >Priority: Trivial > Attachments: FileSystem.java.patch > > > The documentation for FileSystem.createNewFile() indicates that it will > return false if it failed to create the requested file. In fact, it will > only return false if the file existed before the function was called. Also > the function fails to mention that a race condition can result in an > exception be thrown if another process attempts to create the file at the > same time. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5387) Documentation on FileSystem.createNewFile() is misleading
[
https://issues.apache.org/jira/browse/HDFS-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leif Wickland updated HDFS-5387:
Description:
The documentation for FileSystem.createNewFile() indicates that it will return
false if it failed to create the requested file. In fact, it will only return
false if the file existed before the function was called. Also the function
fails to mention that a race condition can result in an exception be thrown if
another process attempts to create the file at the same time.
Here's my suggested improvement to the documentation's wording:
{code}
/**
- * Creates the given Path as a brand-new zero-length file. If
- * create fails, or if it already existed, return false.
- *
+ * Creates the given Path as a brand-new zero-length file.
+ * A race condition can occur inside this function if f is created by another
+ * process after this function has checked for the existence of f. In some
+ * implementations that will result in an exception being thrown.
* @param f path to use for create
+ * @return true if the file was created; false if the path already exists.
+ * @throws IOException
*/
public boolean createNewFile(Path f) throws IOException {
{code}
The same change is attached as a patch.
was:
The documentation for FileSystem.createNewFile() indicates that it will return
false if it failed to create the requested file. In fact, it will only return
false if the file existed before the function was called. Also the function
fails to mention that a race condition can result in an exception be thrown if
another process attempts to create the file at the same time.
Here's my suggested improvement to the documentation's wording:
{code}
/**
- * Creates the given Path as a brand-new zero-length file. If
- * create fails, or if it already existed, return false.
- *
+ * Creates the given Path as a brand-new zero-length file.
+ * A race condition can occur inside this function if f is created by another
+ * process after this function has checked for the existence of f. In some
+ * implementations that will result in an exception being thrown.
* @param f path to use for create
+ * @return true if the file was created; false if the path already exists.
+ * @throws IOException
*/
public boolean createNewFile(Path f) throws IOException {
{code}
> Documentation on FileSystem.createNewFile() is misleading
> -
>
> Key: HDFS-5387
> URL: https://issues.apache.org/jira/browse/HDFS-5387
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Leif Wickland
>Priority: Trivial
> Attachments: FileSystem.java.patch
>
>
> The documentation for FileSystem.createNewFile() indicates that it will
> return false if it failed to create the requested file. In fact, it will
> only return false if the file existed before the function was called. Also
> the function fails to mention that a race condition can result in an
> exception be thrown if another process attempts to create the file at the
> same time.
> Here's my suggested improvement to the documentation's wording:
> {code}
>/**
> - * Creates the given Path as a brand-new zero-length file. If
> - * create fails, or if it already existed, return false.
> - *
> + * Creates the given Path as a brand-new zero-length file.
> + * A race condition can occur inside this function if f is created by
> another
> + * process after this function has checked for the existence of f. In some
> + * implementations that will result in an exception being thrown.
> * @param f path to use for create
> + * @return true if the file was created; false if the path already exists.
> + * @throws IOException
> */
>public boolean createNewFile(Path f) throws IOException {
> {code}
> The same change is attached as a patch.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5387) Documentation on FileSystem.createNewFile() is misleading
[
https://issues.apache.org/jira/browse/HDFS-5387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leif Wickland updated HDFS-5387:
Description:
The documentation for FileSystem.createNewFile() indicates that it will return
false if it failed to create the requested file. In fact, it will only return
false if the file existed before the function was called. Also the function
fails to mention that a race condition can result in an exception be thrown if
another process attempts to create the file at the same time.
Here's my suggested improvement to the documentation's wording:
{code}
/**
- * Creates the given Path as a brand-new zero-length file. If
- * create fails, or if it already existed, return false.
- *
+ * Creates the given Path as a brand-new zero-length file.
+ * A race condition can occur inside this function if f is created by another
+ * process after this function has checked for the existence of f. In some
+ * implementations that will result in an exception being thrown.
* @param f path to use for create
+ * @return true if the file was created; false if the path already exists.
+ * @throws IOException
*/
public boolean createNewFile(Path f) throws IOException {
{code}
was:The documentation for FileSystem.createNewFile() indicates that it will
return false if it failed to create the requested file. In fact, it will only
return false if the file existed before the function was called. Also the
function fails to mention that a race condition can result in an exception be
thrown if another process attempts to create the file at the same time.
> Documentation on FileSystem.createNewFile() is misleading
> -
>
> Key: HDFS-5387
> URL: https://issues.apache.org/jira/browse/HDFS-5387
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Leif Wickland
>Priority: Trivial
> Attachments: FileSystem.java.patch
>
>
> The documentation for FileSystem.createNewFile() indicates that it will
> return false if it failed to create the requested file. In fact, it will
> only return false if the file existed before the function was called. Also
> the function fails to mention that a race condition can result in an
> exception be thrown if another process attempts to create the file at the
> same time.
> Here's my suggested improvement to the documentation's wording:
> {code}
>/**
> - * Creates the given Path as a brand-new zero-length file. If
> - * create fails, or if it already existed, return false.
> - *
> + * Creates the given Path as a brand-new zero-length file.
> + * A race condition can occur inside this function if f is created by
> another
> + * process after this function has checked for the existence of f. In some
> + * implementations that will result in an exception being thrown.
> * @param f path to use for create
> + * @return true if the file was created; false if the path already exists.
> + * @throws IOException
> */
>public boolean createNewFile(Path f) throws IOException {
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799324#comment-13799324 ] Luke Lu commented on HDFS-4885: --- Nicholas suggested offline that we simply pass in the replication number and let the policy impl decided the min rack etc. I agree with him that this is sufficient for replication based block placement (for any hierarchical topology) for now. We'll probably need to introduce a new API to verify RAID block replacement, where replication will always be 1. > Update verifyBlockPlacement() API in BlockPlacementPolicy > - > > Key: HDFS-4885 > URL: https://issues.apache.org/jira/browse/HDFS-4885 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: BlockPlacementPolicy > Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch > > > verifyBlockPlacement() has unused parameter -srcPath as its responsibility > just verify single block rather than files under a specific path. Also the > return value (int) does not make sense as the violation of block placement > has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799334#comment-13799334 ] Tsz Wo (Nicholas), SZE commented on HDFS-4885: -- > Nicholas suggested offline that we simply pass in the replication number ... More details: BlockPlacementPolicy.verifyBlockPlacement(..) should not refer to HdfsFileStatus as the entire block management layer should not refer to namenode specific implementation so that the block management could possibly support other namespace implementations. > We'll probably need to introduce a new API to verify RAID block replacement, > ... For RAID, the facebook implementation use srcPath to determine block placement. So the current API may be good enough. > Update verifyBlockPlacement() API in BlockPlacementPolicy > - > > Key: HDFS-4885 > URL: https://issues.apache.org/jira/browse/HDFS-4885 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: BlockPlacementPolicy > Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch > > > verifyBlockPlacement() has unused parameter -srcPath as its responsibility > just verify single block rather than files under a specific path. Also the > return value (int) does not make sense as the violation of block placement > has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HDFS-5386) Add feature documentation for datanode caching.
[ https://issues.apache.org/jira/browse/HDFS-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-5386: - Assignee: Andrew Wang > Add feature documentation for datanode caching. > --- > > Key: HDFS-5386 > URL: https://issues.apache.org/jira/browse/HDFS-5386 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: HDFS-4949 >Reporter: Chris Nauroth >Assignee: Andrew Wang > > Write feature documentation for datanode caching, covering all of the > following: > * high-level architecture > * OS/native code requirements > * OS configuration (ulimit -l) > * new configuration properties for namenode and datanode > * cache admin CLI commands > * pointers to API for programmatic control of caching directives -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799358#comment-13799358
]
Junping Du commented on HDFS-4885:
--
Thanks Luke and Nicholas for comments!
bq. More details: BlockPlacementPolicy.verifyBlockPlacement(..) should not
refer to HdfsFileStatus as the entire block management layer should not refer
to namenode specific implementation so that the block management could possibly
support other namespace implementations.
+1. It is good to decouple namespace and block management.
bq. For RAID, the facebook implementation use srcPath to determine block
placement. So the current API may be good enough.
so, we only need to change API very slightly (only javadoc and name) that from
{code}
public int verifyBlockPlacement(String srcPath, LocatedBlock lBlk, int minRacks)
{code}
To:
{code}
public int verifyBlockPlacement(String srcPath, LocatedBlock lBlk, int
repNumber)
{code}
Isn't it?
> Update verifyBlockPlacement() API in BlockPlacementPolicy
> -
>
> Key: HDFS-4885
> URL: https://issues.apache.org/jira/browse/HDFS-4885
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Junping Du
> Labels: BlockPlacementPolicy
> Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch
>
>
> verifyBlockPlacement() has unused parameter -srcPath as its responsibility
> just verify single block rather than files under a specific path. Also the
> return value (int) does not make sense as the violation of block placement
> has other case than number of racks, so boolean value should be better.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (HDFS-5365) Fix libhdfs compile error on FreeBSD9
[ https://issues.apache.org/jira/browse/HDFS-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-5365. - Resolution: Fixed Fix Version/s: (was: 2.3.0) 2.2.1 I've merged this to branch-2.2 and updated attribution to release 2.2.1 in CHANGES.txt. Thanks again, Radim. > Fix libhdfs compile error on FreeBSD9 > - > > Key: HDFS-5365 > URL: https://issues.apache.org/jira/browse/HDFS-5365 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Radim Kolar >Assignee: Radim Kolar > Labels: build > Fix For: 3.0.0, 2.2.1 > > Attachments: hdfs-bsd.txt > > > native library do not compiles on freebsd because: > * dlopen is in libc > * limits.h include is missing -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5365) Fix libhdfs compile error on FreeBSD9
[ https://issues.apache.org/jira/browse/HDFS-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799377#comment-13799377 ] Hudson commented on HDFS-5365: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4629 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4629/]) HDFS-5365. Move attribution to 2.2.1 in CHANGES.txt. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533574) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Fix libhdfs compile error on FreeBSD9 > - > > Key: HDFS-5365 > URL: https://issues.apache.org/jira/browse/HDFS-5365 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, libhdfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Radim Kolar >Assignee: Radim Kolar > Labels: build > Fix For: 3.0.0, 2.2.1 > > Attachments: hdfs-bsd.txt > > > native library do not compiles on freebsd because: > * dlopen is in libc > * limits.h include is missing -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799381#comment-13799381 ] Junping Du commented on HDFS-4885: -- Sorry. the return value should be changed to BlockPlacementStatus as v3 patch shows. > Update verifyBlockPlacement() API in BlockPlacementPolicy > - > > Key: HDFS-4885 > URL: https://issues.apache.org/jira/browse/HDFS-4885 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: BlockPlacementPolicy > Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch > > > verifyBlockPlacement() has unused parameter -srcPath as its responsibility > just verify single block rather than files under a specific path. Also the > return value (int) does not make sense as the violation of block placement > has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5388) Loading fsimage fails to find cache pools during namenode startup.
Chris Nauroth created HDFS-5388: --- Summary: Loading fsimage fails to find cache pools during namenode startup. Key: HDFS-5388 URL: https://issues.apache.org/jira/browse/HDFS-5388 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: HDFS-4949 Reporter: Chris Nauroth Assignee: Chris Nauroth The namenode fails during startup due to a check that a cache entry does not refer to a non-existent pool. We need to make a small change to this logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Work started] (HDFS-5388) Loading fsimage fails to find cache pools during namenode startup.
[ https://issues.apache.org/jira/browse/HDFS-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-5388 started by Chris Nauroth. > Loading fsimage fails to find cache pools during namenode startup. > -- > > Key: HDFS-5388 > URL: https://issues.apache.org/jira/browse/HDFS-5388 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5388.1.patch > > > The namenode fails during startup due to a check that a cache entry does not > refer to a non-existent pool. We need to make a small change to this logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5388) Loading fsimage fails to find cache pools during namenode startup.
[ https://issues.apache.org/jira/browse/HDFS-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5388: Attachment: HDFS-5388.1.patch I just noticed the problem this morning after my namenode did a checkpoint and I restarted. This was introduced in HDFS-5096. I'm attaching a fix. After I applied this patch, I was able to load my fsimage and start my namenode successfully. [~cmccabe] and [~andrew.wang], does this look good? > Loading fsimage fails to find cache pools during namenode startup. > -- > > Key: HDFS-5388 > URL: https://issues.apache.org/jira/browse/HDFS-5388 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5388.1.patch > > > The namenode fails during startup due to a check that a cache entry does not > refer to a non-existent pool. We need to make a small change to this logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4376) Fix several race conditions in Balancer and resolve intermittent timeout of TestBalancerWithNodeGroup
[ https://issues.apache.org/jira/browse/HDFS-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799402#comment-13799402 ] Junping Du commented on HDFS-4376: -- Hi [~sureshms], In our offline meet yesterday, you mentioned the test still get failed sometimes. Do you mean it get failed in 2.0 GA or trunk now? As I just fixed race conditions of Balancer in this JIRA this week, so I don't expect it get failed again on trunk (I tried 100 times iterations and no failed locally). It is great that you can provide more details on this? Thx! > Fix several race conditions in Balancer and resolve intermittent timeout of > TestBalancerWithNodeGroup > -- > > Key: HDFS-4376 > URL: https://issues.apache.org/jira/browse/HDFS-4376 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 2.0.3-alpha >Reporter: Aaron T. Myers >Assignee: Junping Du > Fix For: 2.3.0 > > Attachments: BalancerTest-HDFS-4376-v1.tar.gz, HDFS-4376-v1.patch, > HDFS-4376-v2.patch, HDFS-4376-v3.patch, > test-balancer-with-node-group-timeout.txt > > > HDFS-4261 fixed several issues with the balancer and balancer tests, and > reduced the frequency with which TestBalancerWithNodeGroup times out. Despite > this, occasional timeouts still occur in this test. This JIRA is to track and > fix this problem. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4885: - Attachment: HDFS-4885-v4.patch Address above review comments in v4 patch. > Update verifyBlockPlacement() API in BlockPlacementPolicy > - > > Key: HDFS-4885 > URL: https://issues.apache.org/jira/browse/HDFS-4885 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: BlockPlacementPolicy > Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch, > HDFS-4885-v4.patch > > > verifyBlockPlacement() has unused parameter -srcPath as its responsibility > just verify single block rather than files under a specific path. Also the > return value (int) does not make sense as the violation of block placement > has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5203) Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
[
https://issues.apache.org/jira/browse/HDFS-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5203:
Attachment: HDFS-5203.1.patch
I'm attaching a preliminary patch. I've tested this manually. This isn't
ready to commit though, because it doesn't include tests. If anyone wants to
provide feedback while I'm writing tests, I'd appreciate that.
Summary:
* Remove {{CacheManager}} enforcement of uniqueness of descriptors for path +
pool.
* Add API for {{DistributedFileSystem#removePathBasedCacheDescriptors}} given a
{{Path}}. Implement this all the way down through {{DFSClient}}, protobuf,
{{FSNamesystem}}, and {{CacheManager}}.
* Add a new -removeDescriptors command to {{CacheAdmin}}, so that cluster
admins still have an easy way to stop all caching on a path.
* Add a new edit log op for remove-all. I prefer this over possibly sitting in
a tight loop of multiple remove-one ops with the write lock held. Note that I
renumbered some of the ops in this patch, so if you have an existing edit log
from the HDFS-4949 branch, then this is incompatible.
> Concurrent clients that add a cache directive on the same path may
> prematurely uncache from each other.
> ---
>
> Key: HDFS-5203
> URL: https://issues.apache.org/jira/browse/HDFS-5203
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5203.1.patch
>
>
> When a client adds a cache directive, we assign it a unique ID and return
> that ID to the client. If multiple clients add a cache directive for the
> same path, then we return the same ID. If one client then removes the cache
> entry for that ID, then it is removed for all clients. Then, when this
> change becomes visible in subsequent cache reports, the datanodes may
> {{munlock}} the block before the other clients are done with it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5203) Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
[
https://issues.apache.org/jira/browse/HDFS-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799422#comment-13799422
]
Colin Patrick McCabe commented on HDFS-5203:
Thanks for tackling this, Chris.
bq. Remove CacheManager enforcement of uniqueness of descriptors for path +
pool.
great
bq. Add API for DistributedFileSystem#removePathBasedCacheDescriptors given a
Path. Implement this all the way down through DFSClient, protobuf,
FSNamesystem, and CacheManager.
I don't think we need this. Why not just have the shell do a
listPathBasedCacheDescriptors, and then send relevant removes
listPathBasedCacheDescriptors takes an optional path argument so that the
client doesn't have to be bothered with seeing results that don't pertain to
the given path.
We got rid of the batch operations because they were a big headache in general.
i.e. what happens if one remove fails, but the others don't? How can the
client get information about each specific remove that happened? What if the
client has permission to remove one but not the others?
> Concurrent clients that add a cache directive on the same path may
> prematurely uncache from each other.
> ---
>
> Key: HDFS-5203
> URL: https://issues.apache.org/jira/browse/HDFS-5203
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5203.1.patch
>
>
> When a client adds a cache directive, we assign it a unique ID and return
> that ID to the client. If multiple clients add a cache directive for the
> same path, then we return the same ID. If one client then removes the cache
> entry for that ID, then it is removed for all clients. Then, when this
> change becomes visible in subsequent cache reports, the datanodes may
> {{munlock}} the block before the other clients are done with it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5388) Loading fsimage fails to find cache pools during namenode startup.
[ https://issues.apache.org/jira/browse/HDFS-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799473#comment-13799473 ] Colin Patrick McCabe commented on HDFS-5388: +1. thanks, Chris > Loading fsimage fails to find cache pools during namenode startup. > -- > > Key: HDFS-5388 > URL: https://issues.apache.org/jira/browse/HDFS-5388 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Attachments: HDFS-5388.1.patch > > > The namenode fails during startup due to a check that a cache entry does not > refer to a non-existent pool. We need to make a small change to this logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HDFS-5388) Loading fsimage fails to find cache pools during namenode startup.
[ https://issues.apache.org/jira/browse/HDFS-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HDFS-5388. Resolution: Fixed Fix Version/s: HDFS-4949 > Loading fsimage fails to find cache pools during namenode startup. > -- > > Key: HDFS-5388 > URL: https://issues.apache.org/jira/browse/HDFS-5388 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: HDFS-4949 > > Attachments: HDFS-5388.1.patch > > > The namenode fails during startup due to a check that a cache entry does not > refer to a non-existent pool. We need to make a small change to this logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5388) Loading fsimage fails to find cache pools during namenode startup.
[ https://issues.apache.org/jira/browse/HDFS-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5388: --- Issue Type: Sub-task (was: Bug) Parent: HDFS-4949 > Loading fsimage fails to find cache pools during namenode startup. > -- > > Key: HDFS-5388 > URL: https://issues.apache.org/jira/browse/HDFS-5388 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: HDFS-4949 > > Attachments: HDFS-5388.1.patch > > > The namenode fails during startup due to a check that a cache entry does not > refer to a non-existent pool. We need to make a small change to this logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5388) Loading fsimage fails to find cache pools during namenode startup.
[ https://issues.apache.org/jira/browse/HDFS-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799480#comment-13799480 ] Chris Nauroth commented on HDFS-5388: - Thanks for committing, Colin. > Loading fsimage fails to find cache pools during namenode startup. > -- > > Key: HDFS-5388 > URL: https://issues.apache.org/jira/browse/HDFS-5388 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: HDFS-4949 > > Attachments: HDFS-5388.1.patch > > > The namenode fails during startup due to a check that a cache entry does not > refer to a non-existent pool. We need to make a small change to this logic. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-2261) AOP unit tests are not getting compiled or run
[ https://issues.apache.org/jira/browse/HDFS-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799481#comment-13799481 ] Karthik Kambatla commented on HDFS-2261: Just checking again if we have a decision here. I think we should remove these tests or have a concrete plan on enabling them. Otherwise, one call always pull them in from previous branches. > AOP unit tests are not getting compiled or run > --- > > Key: HDFS-2261 > URL: https://issues.apache.org/jira/browse/HDFS-2261 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0-alpha, 2.0.4-alpha > Environment: > https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/834/console > -compile-fault-inject ant target >Reporter: Giridharan Kesavan >Assignee: Karthik Kambatla >Priority: Minor > Attachments: hdfs-2261.patch > > > The tests in src/test/aop are not getting compiled or run. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5203) Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
[
https://issues.apache.org/jira/browse/HDFS-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5203:
Attachment: HDFS-5203.2.patch
bq. We got rid of the batch operations because they were a big headache in
general.
I missed this discussion, but I do think it makes sense. There was a lot of
logic in my first patch around checking the permissions across the union of all
pools. I went down this path initially to prevent a lot of chatter on RPC and
edit log ops, but that might be a premature optimization. We expect far fewer
cache directives than other namesystem objects.
I'm attaching version 2 of the patch with the simpler approach. Tests are
still forthcoming.
> Concurrent clients that add a cache directive on the same path may
> prematurely uncache from each other.
> ---
>
> Key: HDFS-5203
> URL: https://issues.apache.org/jira/browse/HDFS-5203
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5203.1.patch, HDFS-5203.2.patch
>
>
> When a client adds a cache directive, we assign it a unique ID and return
> that ID to the client. If multiple clients add a cache directive for the
> same path, then we return the same ID. If one client then removes the cache
> entry for that ID, then it is removed for all clients. Then, when this
> change becomes visible in subsequent cache reports, the datanodes may
> {{munlock}} the block before the other clients are done with it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5203) Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
[
https://issues.apache.org/jira/browse/HDFS-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799486#comment-13799486
]
Chris Nauroth commented on HDFS-5203:
-
I also meant to mention that I discovered a bug in the path filtering
introduced by the conversion to {{Path}} objects in HDFS-5224. It was trying
to run {{#equals}} between a {{String}} and a {{Path}}, which of course is
always false. I'm planning on rolling a one-liner fix into this patch.
{code}
if (filterPath != null &&
- !directive.getPath().equals(filterPath)) {
+ !directive.getPath().toUri().getPath().equals(filterPath)) {
continue;
{code}
> Concurrent clients that add a cache directive on the same path may
> prematurely uncache from each other.
> ---
>
> Key: HDFS-5203
> URL: https://issues.apache.org/jira/browse/HDFS-5203
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5203.1.patch, HDFS-5203.2.patch
>
>
> When a client adds a cache directive, we assign it a unique ID and return
> that ID to the client. If multiple clients add a cache directive for the
> same path, then we return the same ID. If one client then removes the cache
> entry for that ID, then it is removed for all clients. Then, when this
> change becomes visible in subsequent cache reports, the datanodes may
> {{munlock}} the block before the other clients are done with it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5382) Implement the UI of browsing filesystems in HTML 5 page
[ https://issues.apache.org/jira/browse/HDFS-5382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799496#comment-13799496 ] Jing Zhao commented on HDFS-5382: - bq. Safari, however, still blocks this request, breaking this feature We can see if we can fix/document it later in a separate jira. +1 for the current patch. I will commit this during this weekend or the coming Monday if there is no more comment. > Implement the UI of browsing filesystems in HTML 5 page > --- > > Key: HDFS-5382 > URL: https://issues.apache.org/jira/browse/HDFS-5382 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5382.000.patch, HDFS-5382.001.patch, > HDFS-5382.002.patch > > > The UI of browsing filesystems can be implemented as an HTML 5 application. > The UI can pull the data from WebHDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5203) Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
[
https://issues.apache.org/jira/browse/HDFS-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799497#comment-13799497
]
Colin Patrick McCabe commented on HDFS-5203:
It looks good. The only thing missing is some tests in
{{TestPathBasedCacheRequests}}.
Check out {{testAddRemoveDirectives}} for an example of doing stuff with PBCEs
in a unit test. (and maybe even add it to that function, if you don't want to
create a new test function)
> Concurrent clients that add a cache directive on the same path may
> prematurely uncache from each other.
> ---
>
> Key: HDFS-5203
> URL: https://issues.apache.org/jira/browse/HDFS-5203
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5203.1.patch, HDFS-5203.2.patch
>
>
> When a client adds a cache directive, we assign it a unique ID and return
> that ID to the client. If multiple clients add a cache directive for the
> same path, then we return the same ID. If one client then removes the cache
> entry for that ID, then it is removed for all clients. Then, when this
> change becomes visible in subsequent cache reports, the datanodes may
> {{munlock}} the block before the other clients are done with it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5203) Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
[
https://issues.apache.org/jira/browse/HDFS-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-5203:
---
Issue Type: Sub-task (was: Bug)
Parent: HDFS-4949
> Concurrent clients that add a cache directive on the same path may
> prematurely uncache from each other.
> ---
>
> Key: HDFS-5203
> URL: https://issues.apache.org/jira/browse/HDFS-5203
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5203.1.patch, HDFS-5203.2.patch
>
>
> When a client adds a cache directive, we assign it a unique ID and return
> that ID to the client. If multiple clients add a cache directive for the
> same path, then we return the same ID. If one client then removes the cache
> entry for that ID, then it is removed for all clients. Then, when this
> change becomes visible in subsequent cache reports, the datanodes may
> {{munlock}} the block before the other clients are done with it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5347) add HDFS NFS user guide
[
https://issues.apache.org/jira/browse/HDFS-5347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799509#comment-13799509
]
Jing Zhao commented on HDFS-5347:
-
The document looks great. Some minors and nits:
# "uses" should be "users" in the following
{code}
+ * By default, the export can be mounted by any client. To better control the
access,
+uses can update the following property. The value string contains machine
name and
{code}
# "specifies rw or ro " can be "uses rw or ro to specify read-write or
read-only". Also there should be a space after "networks."
{code}
+characters. Machine name format can be single host, wildcards, and IPv4
networks.The
+access privilege specifies rw or ro access of the machines to exports.If
the access
{code}
# "especially ONCRPC" --> "especially for ONCRPC"?
{code}
+ to add the following. Note, debug trace, especially ONCRPC, can be very
verbose.
{code}
# "trace level" is not consistent with "DEBUG" in the example?
{code}
+To change trace level:
+
+---
+log4j.logger.org.apache.hadoop.hdfs.nfs=DEBUG
+---
{code}
# admin and hdfs here can be "admin" and "hdfs" to make it more clear?
{code}
+ For example, if the NFS client has current user as admin, when the user
accesses
+ the mounted directory, NFS gateway will access HDFS as user admin. To access
HDFS
+ as hdfs user, you must first switch the current user to hdfs on the client
system
{code}
# The following line can be divided into two sentences?
{code}
+ NFS gateway converts UID to user name and HDFS uses username for checking
permissions.
{code}
> add HDFS NFS user guide
> ---
>
> Key: HDFS-5347
> URL: https://issues.apache.org/jira/browse/HDFS-5347
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-5347.001.patch, HDFS-5347.002.patch,
> HdfsNfsGateway.html
>
>
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799521#comment-13799521
]
Hadoop QA commented on HDFS-4885:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609186/HDFS-4885-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5236//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5236//console
This message is automatically generated.
> Update verifyBlockPlacement() API in BlockPlacementPolicy
> -
>
> Key: HDFS-4885
> URL: https://issues.apache.org/jira/browse/HDFS-4885
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Junping Du
> Labels: BlockPlacementPolicy
> Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch,
> HDFS-4885-v4.patch
>
>
> verifyBlockPlacement() has unused parameter -srcPath as its responsibility
> just verify single block rather than files under a specific path. Also the
> return value (int) does not make sense as the violation of block placement
> has other case than number of racks, so boolean value should be better.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5389) Remove INode limitations in Namenode
Lin Xiao created HDFS-5389: -- Summary: Remove INode limitations in Namenode Key: HDFS-5389 URL: https://issues.apache.org/jira/browse/HDFS-5389 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 0.23.1 Reporter: Lin Xiao Priority: Minor Current HDFS Namenode stores all of its metadata in RAM. This has allowed Hadoop clusters to scale to 100K concurrent tasks. However, the memory limits the total number of files that a single NN can store. While Federation allows one to create multiple volumes with additional Namenodes, there is a need to scale a single namespace and also to store multiple namespaces in a single Namenode. When inodes are also stored on persistent storage, the system's boot time can be significantly reduced because there is no need to replay edit logs. It also provides the potential to support extended attributes once the memory size is not the bottleneck. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5389) Remove INode limitations in Namenode
[ https://issues.apache.org/jira/browse/HDFS-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799525#comment-13799525 ] Lin Xiao commented on HDFS-5389: One design to remove the space limits while maintain similar performance is by caching only the working set or hot metadata in Namenode memory. This approach can be very effective because the subset of files that is frequently accessed is much smaller than the full set of files stored in HDFS. This is becoming common because HDFS allows customers to cost effectively store data for many years even though only the latest data is accessed frequently. The goal is that there is only a slight or no degradation when the working set fits in memory. By effective cache replacement policies we believe that we can deal with the cases when applications access cold data occasionally. The current implementation uses log-structured merge-tree(LSM-tree) to store the Namenode's metadata persistently. A related project at CMU, TableFS, showed that LSM-trees were very effective for handling metadata in file systems. We have built a prototype the Namenode using the LevelDB. LevelDB is an open source fast key-value store library using LSM-tree. > Remove INode limitations in Namenode > > > Key: HDFS-5389 > URL: https://issues.apache.org/jira/browse/HDFS-5389 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 0.23.1 >Reporter: Lin Xiao >Priority: Minor > > Current HDFS Namenode stores all of its metadata in RAM. This has allowed > Hadoop clusters to scale to 100K concurrent tasks. However, the memory limits > the total number of files that a single NN can store. While Federation allows > one to create multiple volumes with additional Namenodes, there is a need to > scale a single namespace and also to store multiple namespaces in a single > Namenode. When inodes are also stored on persistent storage, the system's > boot time can be significantly reduced because there is no need to replay > edit logs. It also provides the potential to support extended attributes once > the memory size is not the bottleneck. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5167) Add metrics about the NameNode retry cache
[
https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799536#comment-13799536
]
Akira AJISAKA commented on HDFS-5167:
-
Thanks for taking this issue, [~ozawa].
I built with your patch and found the metrics name was 'rpc'.
The following fix is good to me.
{code}
---registry = new MetricsRegistry("rpc");
+++registry = new MetricsRegistry(name);
{code}
I revised the patch, built, and confirmed the metrics name was
'RetryCacheNamenode'.
> Add metrics about the NameNode retry cache
> --
>
> Key: HDFS-5167
> URL: https://issues.apache.org/jira/browse/HDFS-5167
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, namenode
>Affects Versions: 3.0.0
>Reporter: Jing Zhao
>Assignee: Tsuyoshi OZAWA
>Priority: Minor
> Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch,
> HDFS-5167.4.patch, HDFS-5167.5.patch, HDFS-5167.6.patch, HDFS-5167.6.patch,
> HDFS-5167.7.patch
>
>
> It will be helpful to have metrics in NameNode about the retry cache, such as
> the retry count etc.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5167) Add metrics about the NameNode retry cache
[ https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-5167: Attachment: HDFS-5167.8.patch Attaching the revised patch. > Add metrics about the NameNode retry cache > -- > > Key: HDFS-5167 > URL: https://issues.apache.org/jira/browse/HDFS-5167 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, namenode >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch, > HDFS-5167.4.patch, HDFS-5167.5.patch, HDFS-5167.6.patch, HDFS-5167.6.patch, > HDFS-5167.7.patch, HDFS-5167.8.patch > > > It will be helpful to have metrics in NameNode about the retry cache, such as > the retry count etc. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5203) Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
[
https://issues.apache.org/jira/browse/HDFS-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5203:
Attachment: HDFS-5203.3.patch
Here is patch version 3, which adds tests. There is no difference in the
src/main files since last time.
> Concurrent clients that add a cache directive on the same path may
> prematurely uncache from each other.
> ---
>
> Key: HDFS-5203
> URL: https://issues.apache.org/jira/browse/HDFS-5203
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5203.1.patch, HDFS-5203.2.patch, HDFS-5203.3.patch
>
>
> When a client adds a cache directive, we assign it a unique ID and return
> that ID to the client. If multiple clients add a cache directive for the
> same path, then we return the same ID. If one client then removes the cache
> entry for that ID, then it is removed for all clients. Then, when this
> change becomes visible in subsequent cache reports, the datanodes may
> {{munlock}} the block before the other clients are done with it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5378) In CacheReport, don't send genstamp and length on the wire
[
https://issues.apache.org/jira/browse/HDFS-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799565#comment-13799565
]
Andrew Wang commented on HDFS-5378:
---
Hey Colin, could you rebase this one? Doesn't apply cleanly for me.
> In CacheReport, don't send genstamp and length on the wire
> --
>
> Key: HDFS-5378
> URL: https://issues.apache.org/jira/browse/HDFS-5378
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5378-caching.001.patch, HDFS-5378-caching.002.patch
>
>
> As discussed in HDFS-5096, we don't need genstamp and block length when
> processing cache reports. So let's not send them over the wire (it increases
> the size of cache reports to 3x what it could be).
> Also, we should report the caching statistics alongside the normal DN stats
> in {{StorageReport}}. There's no reason for them to be separate. Since the
> new fields will be optional and default to 0, there will be no extra overhead
> in the non-caching case.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[
https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799572#comment-13799572
]
Tsz Wo (Nicholas), SZE commented on HDFS-4885:
--
- In BlockPlacementStatusDefault, "misReplicatedNum" may be misleading since it
is the number of racks the block should be additionally replicated to. Why
don't we store both requiredRacks and currentRacks?
- hasMisplacement and getMisreplacementDescription may also be misleading since
there may not be any misplacement at all but it does not satisfy the block
placement policy. How about renaming hasMisplacement to "isGoodPlacement" or
"isPlacementPolicySatisfied", and renaming getMisreplacementDescription to
"getErrorDescription" or "getErrorMessage".
- In BlockPlacementPolicyDefault.verifyBlockPlacement(..),
let X = Math.min(2, numberOfReplicas). Then, X <= 2.
We also have numRacks >= 2 since if(numRacks <= 1) return.
As a result, X <= numRacks (i.e. Math.min(X, numRacks) == X) for any case.
So we could simplify the code to
{code}
int minRacks = Math.min(2, numberOfReplicas);
{code}
- Please remove the HdfsFileStatus imports in BlockPlacementPolicy and
BlockPlacementPolicyDefault.
- In NamenodeFsck, let's use targetFileReplication for file.getReplication().
> Update verifyBlockPlacement() API in BlockPlacementPolicy
> -
>
> Key: HDFS-4885
> URL: https://issues.apache.org/jira/browse/HDFS-4885
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Junping Du
>Assignee: Junping Du
> Labels: BlockPlacementPolicy
> Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch,
> HDFS-4885-v4.patch
>
>
> verifyBlockPlacement() has unused parameter -srcPath as its responsibility
> just verify single block rather than files under a specific path. Also the
> return value (int) does not make sense as the violation of block placement
> has other case than number of racks, so boolean value should be better.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (HDFS-5203) Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
[
https://issues.apache.org/jira/browse/HDFS-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe resolved HDFS-5203.
Resolution: Fixed
Fix Version/s: HDFS-4949
> Concurrent clients that add a cache directive on the same path may
> prematurely uncache from each other.
> ---
>
> Key: HDFS-5203
> URL: https://issues.apache.org/jira/browse/HDFS-5203
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: HDFS-4949
>
> Attachments: HDFS-5203.1.patch, HDFS-5203.2.patch, HDFS-5203.3.patch
>
>
> When a client adds a cache directive, we assign it a unique ID and return
> that ID to the client. If multiple clients add a cache directive for the
> same path, then we return the same ID. If one client then removes the cache
> entry for that ID, then it is removed for all clients. Then, when this
> change becomes visible in subsequent cache reports, the datanodes may
> {{munlock}} the block before the other clients are done with it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5203) Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
[
https://issues.apache.org/jira/browse/HDFS-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799584#comment-13799584
]
Colin Patrick McCabe commented on HDFS-5203:
+1, thanks Chris
> Concurrent clients that add a cache directive on the same path may
> prematurely uncache from each other.
> ---
>
> Key: HDFS-5203
> URL: https://issues.apache.org/jira/browse/HDFS-5203
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-5203.1.patch, HDFS-5203.2.patch, HDFS-5203.3.patch
>
>
> When a client adds a cache directive, we assign it a unique ID and return
> that ID to the client. If multiple clients add a cache directive for the
> same path, then we return the same ID. If one client then removes the cache
> entry for that ID, then it is removed for all clients. Then, when this
> change becomes visible in subsequent cache reports, the datanodes may
> {{munlock}} the block before the other clients are done with it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5378) In CacheReport, don't send genstamp and length on the wire
[
https://issues.apache.org/jira/browse/HDFS-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-5378:
---
Attachment: HDFS-5378-caching.003.patch
rebased
> In CacheReport, don't send genstamp and length on the wire
> --
>
> Key: HDFS-5378
> URL: https://issues.apache.org/jira/browse/HDFS-5378
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5378-caching.001.patch,
> HDFS-5378-caching.002.patch, HDFS-5378-caching.003.patch
>
>
> As discussed in HDFS-5096, we don't need genstamp and block length when
> processing cache reports. So let's not send them over the wire (it increases
> the size of cache reports to 3x what it could be).
> Also, we should report the caching statistics alongside the normal DN stats
> in {{StorageReport}}. There's no reason for them to be separate. Since the
> new fields will be optional and default to 0, there will be no extra overhead
> in the non-caching case.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5203) Concurrent clients that add a cache directive on the same path may prematurely uncache from each other.
[
https://issues.apache.org/jira/browse/HDFS-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799595#comment-13799595
]
Chris Nauroth commented on HDFS-5203:
-
Thanks for the code review, Colin.
> Concurrent clients that add a cache directive on the same path may
> prematurely uncache from each other.
> ---
>
> Key: HDFS-5203
> URL: https://issues.apache.org/jira/browse/HDFS-5203
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: HDFS-4949
>
> Attachments: HDFS-5203.1.patch, HDFS-5203.2.patch, HDFS-5203.3.patch
>
>
> When a client adds a cache directive, we assign it a unique ID and return
> that ID to the client. If multiple clients add a cache directive for the
> same path, then we return the same ID. If one client then removes the cache
> entry for that ID, then it is removed for all clients. Then, when this
> change becomes visible in subsequent cache reports, the datanodes may
> {{munlock}} the block before the other clients are done with it.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5326) add modifyDirective to cacheAdmin
[ https://issues.apache.org/jira/browse/HDFS-5326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799606#comment-13799606 ] Colin Patrick McCabe commented on HDFS-5326: I took a whack at this this morning. I ended up reorganizing the types a little bit-- basically creating a type for PBCDs where all the fields were optional (nullable), but continuing to combine that with factories that did validation. It's not quite ready to post but hopefully I'll have it soon. > add modifyDirective to cacheAdmin > - > > Key: HDFS-5326 > URL: https://issues.apache.org/jira/browse/HDFS-5326 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Colin Patrick McCabe > > We should add a way of modifying cache directives on the command-line, > similar to how modifyCachePool works. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HDFS-5180) Add time taken to process the command to audit log
[ https://issues.apache.org/jira/browse/HDFS-5180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita reassigned HDFS-5180: Assignee: Shinichi Yamashita > Add time taken to process the command to audit log > -- > > Key: HDFS-5180 > URL: https://issues.apache.org/jira/browse/HDFS-5180 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0 >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita > Attachments: HDFS-5180.patch > > > Command and ugi are output now by audit log of NameNode. But it is not output > for the processing time of command to audit log. > For example, we must check which command is a problem when a trouble such as > the slow down occurred in NameNode. > It should add the processing time to audit log to know the abnormal sign. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5320) Add datanode caching metrics
[ https://issues.apache.org/jira/browse/HDFS-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-5320: -- Attachment: hdfs-5320-2.patch Here's a new rebased patch, minus the cache report rename since I think it's going away soon. > Add datanode caching metrics > > > Key: HDFS-5320 > URL: https://issues.apache.org/jira/browse/HDFS-5320 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: HDFS-4949 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-5320-1.patch, hdfs-5320-2.patch > > > It'd be good to hook up datanode metrics for # (blocks/bytes) > (cached/uncached/failed to cache) over different time windows > (eternity/1hr/10min/1min). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799625#comment-13799625 ] Colin Patrick McCabe commented on HDFS-5276: I ran mvn eclipse:eclipse locally and it succeeded. The jenkins eclipse:eclipse failure and release audit thing seems to be an environment issue. (looks like a lingering pid file from a previous test run?) Thanks for the +1. Will commit shortly if there are no more comments. > FileSystem.Statistics got performance issue on multi-thread read/write. > --- > > Key: HDFS-5276 > URL: https://issues.apache.org/jira/browse/HDFS-5276 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Chengxiang Li >Assignee: Colin Patrick McCabe > Attachments: DisableFSReadWriteBytesStat.patch, HDFS-5276.001.patch, > HDFS-5276.002.patch, HDFS-5276.003.patch, HDFSStatisticTest.java, > hdfs-test.PNG, jstack-trace.PNG, TestFileSystemStatistics.java, > ThreadLocalStat.patch > > > FileSystem.Statistics is a singleton variable for each FS scheme, each > read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does > not perform well in multi-threads(let's say more than 30 threads). so it may > cause serious performance issue. during our spark test profile, 32 threads > read data from HDFS, about 70% cpu time is spent on > FileSystem.Statistics.incrementBytesRead(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4995) Make getContentSummary() less expensive
[ https://issues.apache.org/jira/browse/HDFS-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4995: - Attachment: HDFS-4995.trunk.patch The new patch moves the lock yield code to the context object. This is based on a casual review comment by Daryn. Having those methods in FSDirectory and FSNamesystem may cause misuses. While moving it into the context class, I added more sanity check to make sure that it has locked the correct locks and only once for each. > Make getContentSummary() less expensive > --- > > Key: HDFS-4995 > URL: https://issues.apache.org/jira/browse/HDFS-4995 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.9, 2.3.0 >Reporter: Kihwal Lee > Attachments: HDFS-4995.trunk1.patch, HDFS-4995.trunk.patch > > > When users call du or count DFS command, getContentSummary() method is called > against namenode. If the directory has many directories and files, it could > hold the namesystem lock for a long time. We've seen it taking over 20 > seconds. Namenode should not allow regular users to cause extended locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4995) Make getContentSummary() less expensive
[ https://issues.apache.org/jira/browse/HDFS-4995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799628#comment-13799628 ] Kihwal Lee commented on HDFS-4995: -- I've tested this with 100 threads creating, renaming deleting and also reader client calling getContentSummary(). The threshold was set to 2 in this test, so that it does lock yielding very frequently. > Make getContentSummary() less expensive > --- > > Key: HDFS-4995 > URL: https://issues.apache.org/jira/browse/HDFS-4995 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.9, 2.3.0 >Reporter: Kihwal Lee > Attachments: HDFS-4995.trunk1.patch, HDFS-4995.trunk.patch > > > When users call du or count DFS command, getContentSummary() method is called > against namenode. If the directory has many directories and files, it could > hold the namesystem lock for a long time. We've seen it taking over 20 > seconds. Namenode should not allow regular users to cause extended locking. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-2832) Heartbeats from Datandode should include one storage report per storage
[ https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-2832: Summary: Heartbeats from Datandode should include one storage report per storage (was: Enable support for heterogeneous storages in HDFS) > Heartbeats from Datandode should include one storage report per storage > --- > > Key: HDFS-2832 > URL: https://issues.apache.org/jira/browse/HDFS-2832 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: 20130813-HeterogeneousStorage.pdf > > > HDFS currently supports configuration where storages are a list of > directories. Typically each of these directories correspond to a volume with > its own file system. All these directories are homogeneous and therefore > identified as a single storage at the namenode. I propose, change to the > current model where Datanode * is a * storage, to Datanode * is a collection > * of strorages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5390) Incremental block reports should be sent per storage
Arpit Agarwal created HDFS-5390: --- Summary: Incremental block reports should be sent per storage Key: HDFS-5390 URL: https://issues.apache.org/jira/browse/HDFS-5390 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: Heterogeneous Storage (HDFS-2832) Reporter: Arpit Agarwal Assignee: Arpit Agarwal Incremental block reports should be sent per storage since the NameNode now expects all block reports to be sent per storage. Currently the Datanode tracks pending incremental block reports for all storages in a single queue. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5390) Incremental block reports should be sent per storage
[ https://issues.apache.org/jira/browse/HDFS-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5390: Description: Incremental block reports should be sent per storage directory since the NameNode now expects all block reports to be sent per storage. Currently the Datanode tracks pending incremental block reports for all storages in a single queue. This is a continuation of the work started by HDFS-5134 and HDFS-4988 to split block processing per storage directory. was:Incremental block reports should be sent per storage since the NameNode now expects all block reports to be sent per storage. Currently the Datanode tracks pending incremental block reports for all storages in a single queue. > Incremental block reports should be sent per storage > > > Key: HDFS-5390 > URL: https://issues.apache.org/jira/browse/HDFS-5390 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Incremental block reports should be sent per storage directory since the > NameNode now expects all block reports to be sent per storage. Currently the > Datanode tracks pending incremental block reports for all storages in a > single queue. > This is a continuation of the work started by HDFS-5134 and HDFS-4988 to > split block processing per storage directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5390) Incremental block reports should be sent per storage directory
[ https://issues.apache.org/jira/browse/HDFS-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5390: Summary: Incremental block reports should be sent per storage directory (was: Incremental block reports should be sent per storage) > Incremental block reports should be sent per storage directory > -- > > Key: HDFS-5390 > URL: https://issues.apache.org/jira/browse/HDFS-5390 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Incremental block reports should be sent per storage directory since the > NameNode now expects all block reports to be sent per storage. Currently the > Datanode tracks pending incremental block reports for all storages in a > single queue. > This is a continuation of the work started by HDFS-5134 and HDFS-4988 to > split block processing per storage directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-2832: Summary: Enable support for heterogeneous storages in HDFS (was: Heartbeats from Datandode should include one storage report per storage) > Enable support for heterogeneous storages in HDFS > - > > Key: HDFS-2832 > URL: https://issues.apache.org/jira/browse/HDFS-2832 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: 20130813-HeterogeneousStorage.pdf > > > HDFS currently supports configuration where storages are a list of > directories. Typically each of these directories correspond to a volume with > its own file system. All these directories are homogeneous and therefore > identified as a single storage at the namenode. I propose, change to the > current model where Datanode * is a * storage, to Datanode * is a collection > * of strorages. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5377) Heartbeats from Datandode should include one storage report per storage
[ https://issues.apache.org/jira/browse/HDFS-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5377: Summary: Heartbeats from Datandode should include one storage report per storage (was: Storage usage information not reflected in block manager state) > Heartbeats from Datandode should include one storage report per storage > --- > > Key: HDFS-5377 > URL: https://issues.apache.org/jira/browse/HDFS-5377 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Per-storage disk usage data is not being reported to the block manager. Need > to debug before I know more. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5377) Heartbeats from Datandode should include one storage report per storage
[ https://issues.apache.org/jira/browse/HDFS-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5377: Description: Heartbeats must include one storage report per storage directory, so that usage information for each directory is available to the DataNode for block placement. This is a continuation of the work started by HDFS-5134 and HDFS-4988 to split block processing per storage directory. was:Per-storage disk usage data is not being reported to the block manager. Need to debug before I know more. > Heartbeats from Datandode should include one storage report per storage > --- > > Key: HDFS-5377 > URL: https://issues.apache.org/jira/browse/HDFS-5377 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Heartbeats must include one storage report per storage directory, so that > usage information for each directory is available to the DataNode for block > placement. > This is a continuation of the work started by HDFS-5134 and HDFS-4988 to > split block processing per storage directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5377) Heartbeats from Datandode should include one storage report per storage directory
[ https://issues.apache.org/jira/browse/HDFS-5377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5377: Summary: Heartbeats from Datandode should include one storage report per storage directory (was: Heartbeats from Datandode should include one storage report per storage) > Heartbeats from Datandode should include one storage report per storage > directory > - > > Key: HDFS-5377 > URL: https://issues.apache.org/jira/browse/HDFS-5377 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Heterogeneous Storage (HDFS-2832) >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Heartbeats must include one storage report per storage directory, so that > usage information for each directory is available to the DataNode for block > placement. > This is a continuation of the work started by HDFS-5134 and HDFS-4988 to > split block processing per storage directory. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5378) In CacheReport, don't send genstamp and length on the wire
[
https://issues.apache.org/jira/browse/HDFS-5378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799640#comment-13799640
]
Andrew Wang commented on HDFS-5378:
---
Looks mostly good, just a few things:
* Need to fixup some compile errors in TestFsDatasetCache
* Based on the actual contents of {{StorageReport}} it seems okay to just cram
everything in there even though some things aren't per-storage. I'd just do the
same fields as for disk: capacity, used, remaining, block pool used.
* I don't think we actually track per-blockpool cache usage, so need to add
that. This is one of the things I thought of related to federation.
> In CacheReport, don't send genstamp and length on the wire
> --
>
> Key: HDFS-5378
> URL: https://issues.apache.org/jira/browse/HDFS-5378
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5378-caching.001.patch,
> HDFS-5378-caching.002.patch, HDFS-5378-caching.003.patch
>
>
> As discussed in HDFS-5096, we don't need genstamp and block length when
> processing cache reports. So let's not send them over the wire (it increases
> the size of cache reports to 3x what it could be).
> Also, we should report the caching statistics alongside the normal DN stats
> in {{StorageReport}}. There's no reason for them to be separate. Since the
> new fields will be optional and default to 0, there will be no extra overhead
> in the non-caching case.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5347) add HDFS NFS user guide
[ https://issues.apache.org/jira/browse/HDFS-5347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5347: - Attachment: HDFS-5347.003.patch Thank you, Jing. I've uploaded a new patch to address the comments. > add HDFS NFS user guide > --- > > Key: HDFS-5347 > URL: https://issues.apache.org/jira/browse/HDFS-5347 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5347.001.patch, HDFS-5347.002.patch, > HDFS-5347.003.patch, HdfsNfsGateway.html > > -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799651#comment-13799651 ] Junping Du commented on HDFS-4885: -- Thanks Nicholas for carefully review. All comments here make sense to me. Some name issues like misplacement is just trying to be consistent with MisReplicatedBlocks in existing NamenodeFsck. Incorporate all comments in v5 patch. > Update verifyBlockPlacement() API in BlockPlacementPolicy > - > > Key: HDFS-4885 > URL: https://issues.apache.org/jira/browse/HDFS-4885 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: BlockPlacementPolicy > Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch, > HDFS-4885-v4.patch, HDFS-4885-v5.patch > > > verifyBlockPlacement() has unused parameter -srcPath as its responsibility > just verify single block rather than files under a specific path. Also the > return value (int) does not make sense as the violation of block placement > has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-4885) Update verifyBlockPlacement() API in BlockPlacementPolicy
[ https://issues.apache.org/jira/browse/HDFS-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4885: - Attachment: HDFS-4885-v5.patch > Update verifyBlockPlacement() API in BlockPlacementPolicy > - > > Key: HDFS-4885 > URL: https://issues.apache.org/jira/browse/HDFS-4885 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Junping Du >Assignee: Junping Du > Labels: BlockPlacementPolicy > Attachments: HDFS-4885.patch, HDFS-4885-v2.patch, HDFS-4885-v3.patch, > HDFS-4885-v4.patch, HDFS-4885-v5.patch > > > verifyBlockPlacement() has unused parameter -srcPath as its responsibility > just verify single block rather than files under a specific path. Also the > return value (int) does not make sense as the violation of block placement > has other case than number of racks, so boolean value should be better. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5167) Add metrics about the NameNode retry cache
[
https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799676#comment-13799676
]
Hadoop QA commented on HDFS-5167:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12609215/HDFS-5167.8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5237//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5237//console
This message is automatically generated.
> Add metrics about the NameNode retry cache
> --
>
> Key: HDFS-5167
> URL: https://issues.apache.org/jira/browse/HDFS-5167
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, namenode
>Affects Versions: 3.0.0
>Reporter: Jing Zhao
>Assignee: Tsuyoshi OZAWA
>Priority: Minor
> Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch,
> HDFS-5167.4.patch, HDFS-5167.5.patch, HDFS-5167.6.patch, HDFS-5167.6.patch,
> HDFS-5167.7.patch, HDFS-5167.8.patch
>
>
> It will be helpful to have metrics in NameNode about the retry cache, such as
> the retry count etc.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5276: --- Resolution: Fixed Fix Version/s: 2.3.0 Status: Resolved (was: Patch Available) > FileSystem.Statistics got performance issue on multi-thread read/write. > --- > > Key: HDFS-5276 > URL: https://issues.apache.org/jira/browse/HDFS-5276 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Chengxiang Li >Assignee: Colin Patrick McCabe > Fix For: 2.3.0 > > Attachments: DisableFSReadWriteBytesStat.patch, HDFS-5276.001.patch, > HDFS-5276.002.patch, HDFS-5276.003.patch, HDFSStatisticTest.java, > hdfs-test.PNG, jstack-trace.PNG, TestFileSystemStatistics.java, > ThreadLocalStat.patch > > > FileSystem.Statistics is a singleton variable for each FS scheme, each > read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does > not perform well in multi-threads(let's say more than 30 threads). so it may > cause serious performance issue. during our spark test profile, 32 threads > read data from HDFS, about 70% cpu time is spent on > FileSystem.Statistics.incrementBytesRead(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799689#comment-13799689 ] Hudson commented on HDFS-5276: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4632 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4632/]) HDFS-5276. FileSystem.Statistics should use thread-local counters to avoid multi-threaded performance issues on read/write. (Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1533668) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/FCStatisticsBaseTest.java > FileSystem.Statistics got performance issue on multi-thread read/write. > --- > > Key: HDFS-5276 > URL: https://issues.apache.org/jira/browse/HDFS-5276 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Chengxiang Li >Assignee: Colin Patrick McCabe > Fix For: 2.3.0 > > Attachments: DisableFSReadWriteBytesStat.patch, HDFS-5276.001.patch, > HDFS-5276.002.patch, HDFS-5276.003.patch, HDFSStatisticTest.java, > hdfs-test.PNG, jstack-trace.PNG, TestFileSystemStatistics.java, > ThreadLocalStat.patch > > > FileSystem.Statistics is a singleton variable for each FS scheme, each > read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does > not perform well in multi-threads(let's say more than 30 threads). so it may > cause serious performance issue. during our spark test profile, 32 threads > read data from HDFS, about 70% cpu time is spent on > FileSystem.Statistics.incrementBytesRead(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5167) Add metrics about the NameNode retry cache
[ https://issues.apache.org/jira/browse/HDFS-5167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13799692#comment-13799692 ] Tsuyoshi OZAWA commented on HDFS-5167: -- Thanks for catching up, Akira. [~jingzhao], please let me know if you have additional comments. > Add metrics about the NameNode retry cache > -- > > Key: HDFS-5167 > URL: https://issues.apache.org/jira/browse/HDFS-5167 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, namenode >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Tsuyoshi OZAWA >Priority: Minor > Attachments: HDFS-5167.1.patch, HDFS-5167.2.patch, HDFS-5167.3.patch, > HDFS-5167.4.patch, HDFS-5167.5.patch, HDFS-5167.6.patch, HDFS-5167.6.patch, > HDFS-5167.7.patch, HDFS-5167.8.patch > > > It will be helpful to have metrics in NameNode about the retry cache, such as > the retry count etc. -- This message was sent by Atlassian JIRA (v6.1#6144)
