[jira] [Commented] (HDFS-5346) Replication queues should not be initialized in the middle of IBR processing.
[
https://issues.apache.org/jira/browse/HDFS-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794933#comment-13794933
]
Hadoop QA commented on HDFS-5346:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608429/HDFS-5346.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5190//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5190//console
This message is automatically generated.
> Replication queues should not be initialized in the middle of IBR processing.
> -
>
> Key: HDFS-5346
> URL: https://issues.apache.org/jira/browse/HDFS-5346
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, performance
>Affects Versions: 0.23.9, 2.3.0
>Reporter: Kihwal Lee
>Assignee: Ravi Prakash
> Fix For: 2.3.0, 0.23.10
>
> Attachments: HDFS-5346.patch
>
>
> When initial block reports are being processed, checkMode() is called from
> incrementSafeBlockCount(). This causes the replication queues to be
> initialized in the middle of processing a block report in the IBR processing
> mode. If there are many block reports waiting to be processed,
> SafeModeMonitor won't be able to make name node leave the safe mode soon. It
> appears that the block report processing speed degrades considerably during
> this time.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5361) Change the unit of StartupProgress 'PercentComplete' to percentage
[ https://issues.apache.org/jira/browse/HDFS-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794918#comment-13794918 ] Suresh Srinivas commented on HDFS-5361: --- Given this is incompatible, the target version for this cannot be 2.3.0. It can only be done in 3.0. Alternative is to add another metrics with a different name and leave the current one as is. > Change the unit of StartupProgress 'PercentComplete' to percentage > -- > > Key: HDFS-5361 > URL: https://issues.apache.org/jira/browse/HDFS-5361 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.1.0-beta >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Minor > Labels: metrics, newbie > Attachments: HDFS-5361.2.patch, HDFS-5361.patch > > > Now the unit of 'PercentComplete' metrics is rate (maximum is 1.0). It's > confusing for users because its name includes "percent". > The metrics should be multiplied by 100. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5330) fix readdir and readdirplus for large directories
[
https://issues.apache.org/jira/browse/HDFS-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794917#comment-13794917
]
Hadoop QA commented on HDFS-5330:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608432/HDFS-5330.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs-nfs:
org.apache.hadoop.hdfs.nfs.TestReaddir
org.apache.hadoop.hdfs.nfs.nfs3.TestWrites
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5193//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5193//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5193//console
This message is automatically generated.
> fix readdir and readdirplus for large directories
> -
>
> Key: HDFS-5330
> URL: https://issues.apache.org/jira/browse/HDFS-5330
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: nfs
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-5330.001.patch
>
>
> These two calls need to use cookies to do multiple round trips to namenode to
> get the complete list of the dirents. Currently implementation passes an
> inode path as "startAfter" for listPath(), however, namenode doesn't resolve
> startAfter as an inode path. Better use file name as "startAfter".
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5364) Add OpenFileCtx cache
Brandon Li created HDFS-5364: Summary: Add OpenFileCtx cache Key: HDFS-5364 URL: https://issues.apache.org/jira/browse/HDFS-5364 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Brandon Li Assignee: Brandon Li NFS gateway can run out of memory when the stream timeout is set to a relatively long period(e.g., >1 minute) and user uploads thousands of files in parallel. Each stream DFSClient creates a DataStreamer thread, and will eventually run out of memory by creating too many threads. NFS gateway should have a OpenFileCtx cache to limit the total opened files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5361) Change the unit of StartupProgress 'PercentComplete' to percentage
[
https://issues.apache.org/jira/browse/HDFS-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794907#comment-13794907
]
Hadoop QA commented on HDFS-5361:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608433/HDFS-5361.2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5192//console
This message is automatically generated.
> Change the unit of StartupProgress 'PercentComplete' to percentage
> --
>
> Key: HDFS-5361
> URL: https://issues.apache.org/jira/browse/HDFS-5361
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.1.0-beta
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: metrics, newbie
> Attachments: HDFS-5361.2.patch, HDFS-5361.patch
>
>
> Now the unit of 'PercentComplete' metrics is rate (maximum is 1.0). It's
> confusing for users because its name includes "percent".
> The metrics should be multiplied by 100.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sathish updated HDFS-5343: -- Attachment: HDFS-5343-001.patch > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > Attachments: HDFS-5343-001.patch > > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[
https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794881#comment-13794881
]
sathish commented on HDFS-5343:
---
// update current position
if (updatePosition) {
pos = offset;
blockEnd = blk.getStartOffset() + blk.getBlockSize() - 1;
currentLocatedBlock = blk;
}
Yes,the above check is the cause for reading unexpexcted bytes,when reading
from snapshot file,and one more thing is the cat command uses the readstrategy
method to read the bytes from a file,in this it taking block end as length to
read the content in the file,so even if we snapshot file also,it is not taking
lenght as indication to read content in snapshot files,
private int readWithStrategy(ReaderStrategy strategy, int off, int len) throws
IOException {
dfsClient.checkOpen();
if (closed) {
throw new IOException("Stream closed");
}
Map> corruptedBlockMap
= new HashMap>();
failures = 0;
if (pos < getFileLength()) {
int retries = 2;
while (retries > 0) {
try {
// currentNode can be left as null if previous read had a checksum
// error on the same block. See HDFS-3067
if (pos > blockEnd || currentNode == null) {
currentNode = blockSeekTo(pos);
}
int realLen = (int) Math.min(len, (blockEnd - pos + 1L));
int result = readBuffer(strategy, off, realLen, corruptedBlockMap);
here it is taking blockend as indication to read the content in that file,so we
can change that blockend for that snapshot files to filelength as like this
// update current position
if (updatePosition) {
pos = offset;
blockEnd = Math.min((locatedBlocks.getFileLength() - 1),
(blk.getStartOffset()
+ blk.getBlockSize() - 1));
currentLocatedBlock = blk;
}
return blk;
}
> When cat command is issued on snapshot files getting unexpected result
> --
>
> Key: HDFS-5343
> URL: https://issues.apache.org/jira/browse/HDFS-5343
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Reporter: sathish
>Assignee: sathish
>
> first if we create one file with some file length and take the snapshot of
> that file,and again append some data through append method to that file,then
> if we do cat command operation on snapshot of that file,in general it should
> dispaly the data what we added with create operation,but it is displaying the
> total data i.e. create +_ appended data.
> but if we do the same operation and if we read the contents of snapshot file
> through input stream it is just displaying the data created in snapshoted
> files.
> in this the behaviour of cat command and reading through inputstream is
> getting different
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794883#comment-13794883 ] sathish commented on HDFS-5343: --- for this i am attching a fix and testcase in this pacth,please review it > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5331) make SnapshotDiff.java to a o.a.h.util.Tool interface implementation
[ https://issues.apache.org/jira/browse/HDFS-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5331: Status: Patch Available (was: Open) > make SnapshotDiff.java to a o.a.h.util.Tool interface implementation > > > Key: HDFS-5331 > URL: https://issues.apache.org/jira/browse/HDFS-5331 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 2.1.1-beta, 3.0.0 >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5331.patch > > > SnapshotDiff.java is a plain class with main() method now. > Convert it to o.a.h.util.Tool interface implementation for better look and > usage in tests in future, -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5331) make SnapshotDiff.java to a o.a.h.util.Tool interface implementation
[ https://issues.apache.org/jira/browse/HDFS-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794877#comment-13794877 ] Vinay commented on HDFS-5331: - Thanks Uma for taking a look at Jira. Since comments are already covered in attached patch making to patch available status back > make SnapshotDiff.java to a o.a.h.util.Tool interface implementation > > > Key: HDFS-5331 > URL: https://issues.apache.org/jira/browse/HDFS-5331 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0, 2.1.1-beta >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5331.patch > > > SnapshotDiff.java is a plain class with main() method now. > Convert it to o.a.h.util.Tool interface implementation for better look and > usage in tests in future, -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5283: Attachment: (was: HDFS-5283.patch) > NN not coming out of startup safemode due to under construction blocks only > inside snapshots also counted in safemode threshhold > > > Key: HDFS-5283 > URL: https://issues.apache.org/jira/browse/HDFS-5283 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0, 2.1.1-beta >Reporter: Vinay >Assignee: Vinay >Priority: Blocker > Attachments: HDFS-5283.000.patch, HDFS-5283.patch, HDFS-5283.patch, > HDFS-5283.patch, HDFS-5283.patch > > > This is observed in one of our env: > 1. A MR Job was running which has created some temporary files and was > writing to them. > 2. Snapshot was taken > 3. And Job was killed and temporary files were deleted. > 4. Namenode restarted. > 5. After restart Namenode was in safemode waiting for blocks > Analysis > - > 1. Since the snapshot taken also includes the temporary files which were > open, and later original files are deleted. > 2. UnderConstruction blocks count was taken from leases. not considered the > UC blocks only inside snapshots > 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5283: Attachment: HDFS-5283.patch > NN not coming out of startup safemode due to under construction blocks only > inside snapshots also counted in safemode threshhold > > > Key: HDFS-5283 > URL: https://issues.apache.org/jira/browse/HDFS-5283 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0, 2.1.1-beta >Reporter: Vinay >Assignee: Vinay >Priority: Blocker > Attachments: HDFS-5283.000.patch, HDFS-5283.patch, HDFS-5283.patch, > HDFS-5283.patch, HDFS-5283.patch, HDFS-5283.patch > > > This is observed in one of our env: > 1. A MR Job was running which has created some temporary files and was > writing to them. > 2. Snapshot was taken > 3. And Job was killed and temporary files were deleted. > 4. Namenode restarted. > 5. After restart Namenode was in safemode waiting for blocks > Analysis > - > 1. Since the snapshot taken also includes the temporary files which were > open, and later original files are deleted. > 2. UnderConstruction blocks count was taken from leases. not considered the > UC blocks only inside snapshots > 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5283) NN not coming out of startup safemode due to under construction blocks only inside snapshots also counted in safemode threshhold
[ https://issues.apache.org/jira/browse/HDFS-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-5283: Attachment: HDFS-5283.patch Thanks Nicholas for taking a look at patch. Attaching the patch updated with as per comments. Please review. Thanks > NN not coming out of startup safemode due to under construction blocks only > inside snapshots also counted in safemode threshhold > > > Key: HDFS-5283 > URL: https://issues.apache.org/jira/browse/HDFS-5283 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0, 2.1.1-beta >Reporter: Vinay >Assignee: Vinay >Priority: Blocker > Attachments: HDFS-5283.000.patch, HDFS-5283.patch, HDFS-5283.patch, > HDFS-5283.patch, HDFS-5283.patch > > > This is observed in one of our env: > 1. A MR Job was running which has created some temporary files and was > writing to them. > 2. Snapshot was taken > 3. And Job was killed and temporary files were deleted. > 4. Namenode restarted. > 5. After restart Namenode was in safemode waiting for blocks > Analysis > - > 1. Since the snapshot taken also includes the temporary files which were > open, and later original files are deleted. > 2. UnderConstruction blocks count was taken from leases. not considered the > UC blocks only inside snapshots > 3. So safemode threshold count was more and NN did not come out of safemode -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5363) Create SPENGO-authenticated connection in URLConnectionFactory instead WebHdfsFileSystem
[
https://issues.apache.org/jira/browse/HDFS-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794872#comment-13794872
]
Hadoop QA commented on HDFS-5363:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608405/HDFS-5363.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5189//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5189//console
This message is automatically generated.
> Create SPENGO-authenticated connection in URLConnectionFactory instead
> WebHdfsFileSystem
>
>
> Key: HDFS-5363
> URL: https://issues.apache.org/jira/browse/HDFS-5363
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5363.000.patch
>
>
> Currently the WebHdfsSystem class creates the http connection of urls that
> require SPENGO authentication. This patch moves the above logic to
> URLConnectionFactory, which is the factory class that supposes to create all
> http connection of WebHdfs client.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5361) Change the unit of StartupProgress 'PercentComplete' to percentage
[ https://issues.apache.org/jira/browse/HDFS-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-5361: Attachment: HDFS-5361.2.patch Modified the patch to fix unit tests. > Change the unit of StartupProgress 'PercentComplete' to percentage > -- > > Key: HDFS-5361 > URL: https://issues.apache.org/jira/browse/HDFS-5361 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.1.0-beta >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Minor > Labels: metrics, newbie > Attachments: HDFS-5361.2.patch, HDFS-5361.patch > > > Now the unit of 'PercentComplete' metrics is rate (maximum is 1.0). It's > confusing for users because its name includes "percent". > The metrics should be multiplied by 100. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5330) fix readdir and readdirplus for large directories
[ https://issues.apache.org/jira/browse/HDFS-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5330: - Status: Patch Available (was: Open) > fix readdir and readdirplus for large directories > - > > Key: HDFS-5330 > URL: https://issues.apache.org/jira/browse/HDFS-5330 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5330.001.patch > > > These two calls need to use cookies to do multiple round trips to namenode to > get the complete list of the dirents. Currently implementation passes an > inode path as "startAfter" for listPath(), however, namenode doesn't resolve > startAfter as an inode path. Better use file name as "startAfter". -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5330) fix readdir and readdirplus for large directories
[ https://issues.apache.org/jira/browse/HDFS-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-5330: - Attachment: HDFS-5330.001.patch > fix readdir and readdirplus for large directories > - > > Key: HDFS-5330 > URL: https://issues.apache.org/jira/browse/HDFS-5330 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-5330.001.patch > > > These two calls need to use cookies to do multiple round trips to namenode to > get the complete list of the dirents. Currently implementation passes an > inode path as "startAfter" for listPath(), however, namenode doesn't resolve > startAfter as an inode path. Better use file name as "startAfter". -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5346) Replication queues should not be initialized in the middle of IBR processing.
[ https://issues.apache.org/jira/browse/HDFS-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HDFS-5346: --- Attachment: HDFS-5346.patch Patch for trunk > Replication queues should not be initialized in the middle of IBR processing. > - > > Key: HDFS-5346 > URL: https://issues.apache.org/jira/browse/HDFS-5346 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, performance >Affects Versions: 0.23.9, 2.3.0 >Reporter: Kihwal Lee >Assignee: Ravi Prakash > Fix For: 2.3.0, 0.23.10 > > Attachments: HDFS-5346.patch > > > When initial block reports are being processed, checkMode() is called from > incrementSafeBlockCount(). This causes the replication queues to be > initialized in the middle of processing a block report in the IBR processing > mode. If there are many block reports waiting to be processed, > SafeModeMonitor won't be able to make name node leave the safe mode soon. It > appears that the block report processing speed degrades considerably during > this time. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5346) Replication queues should not be initialized in the middle of IBR processing.
[ https://issues.apache.org/jira/browse/HDFS-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HDFS-5346: --- Status: Patch Available (was: Open) > Replication queues should not be initialized in the middle of IBR processing. > - > > Key: HDFS-5346 > URL: https://issues.apache.org/jira/browse/HDFS-5346 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, performance >Affects Versions: 0.23.9, 2.3.0 >Reporter: Kihwal Lee >Assignee: Ravi Prakash > Fix For: 2.3.0, 0.23.10 > > Attachments: HDFS-5346.patch > > > When initial block reports are being processed, checkMode() is called from > incrementSafeBlockCount(). This causes the replication queues to be > initialized in the middle of processing a block report in the IBR processing > mode. If there are many block reports waiting to be processed, > SafeModeMonitor won't be able to make name node leave the safe mode soon. It > appears that the block report processing speed degrades considerably during > this time. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5361) Change the unit of StartupProgress 'PercentComplete' to percentage
[
https://issues.apache.org/jira/browse/HDFS-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794801#comment-13794801
]
Hadoop QA commented on HDFS-5361:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608387/HDFS-5361.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestNameNodeJspHelper
org.apache.hadoop.hdfs.server.namenode.TestStartupProgressServlet
org.apache.hadoop.hdfs.server.namenode.startupprogress.TestStartupProgress
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5187//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5187//console
This message is automatically generated.
> Change the unit of StartupProgress 'PercentComplete' to percentage
> --
>
> Key: HDFS-5361
> URL: https://issues.apache.org/jira/browse/HDFS-5361
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.1.0-beta
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: metrics, newbie
> Attachments: HDFS-5361.patch
>
>
> Now the unit of 'PercentComplete' metrics is rate (maximum is 1.0). It's
> confusing for users because its name includes "percent".
> The metrics should be multiplied by 100.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[
https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794772#comment-13794772
]
Hadoop QA commented on HDFS-5276:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608402/HDFS-5276.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5188//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5188//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5188//console
This message is automatically generated.
> FileSystem.Statistics got performance issue on multi-thread read/write.
> ---
>
> Key: HDFS-5276
> URL: https://issues.apache.org/jira/browse/HDFS-5276
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.4-alpha
>Reporter: Chengxiang Li
>Assignee: Colin Patrick McCabe
> Attachments: DisableFSReadWriteBytesStat.patch, HDFS-5276.001.patch,
> HDFS-5276.002.patch, HDFS-5276.003.patch, HDFSStatisticTest.java,
> hdfs-test.PNG, jstack-trace.PNG, TestFileSystemStatistics.java,
> ThreadLocalStat.patch
>
>
> FileSystem.Statistics is a singleton variable for each FS scheme, each
> read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does
> not perform well in multi-threads(let's say more than 30 threads). so it may
> cause serious performance issue. during our spark test profile, 32 threads
> read data from HDFS, about 70% cpu time is spent on
> FileSystem.Statistics.incrementBytesRead().
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5096) Automatically cache new data added to a cached path
[ https://issues.apache.org/jira/browse/HDFS-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5096: --- Attachment: HDFS-5096-caching.009.patch here's an updated patch which is rebased on the subtasks already committed, and includes most of the feedback here (still are a few things I'm working on) > Automatically cache new data added to a cached path > --- > > Key: HDFS-5096 > URL: https://issues.apache.org/jira/browse/HDFS-5096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Andrew Wang >Assignee: Colin Patrick McCabe > Attachments: HDFS-5096-caching.005.patch, > HDFS-5096-caching.006.patch, HDFS-5096-caching.009.patch > > > For some applications, it's convenient to specify a path to cache, and have > HDFS automatically cache new data added to the path without sending a new > caching request or a manual refresh command. > One example is new data appended to a cached file. It would be nice to > re-cache a block at the new appended length, and cache new blocks added to > the file. > Another example is a cached Hive partition directory, where a user can drop > new files directly into the partition. It would be nice if these new files > were cached. > In both cases, this automatic caching would happen after the file is closed, > i.e. block replica is finalized. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5352) Server#initLog() doesn't close InputStream in httpfs
[
https://issues.apache.org/jira/browse/HDFS-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794756#comment-13794756
]
Hudson commented on HDFS-5352:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4605 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4605/])
HDFS-5352. Server#initLog() doesn't close InputStream in httpfs. Contributed by
Ted Yu. (jing9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1532158)
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Server#initLog() doesn't close InputStream in httpfs
>
>
> Key: HDFS-5352
> URL: https://issues.apache.org/jira/browse/HDFS-5352
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.3.0
>
> Attachments: hdfs-5352.patch
>
>
> Here is related code snippet in
> hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java:
> {code}
> Properties props = new Properties();
> try {
> InputStream is = getResource(DEFAULT_LOG4J_PROPERTIES);
> props.load(is);
> } catch (IOException ex) {
> throw new ServerException(ServerException.ERROR.S03,
> DEFAULT_LOG4J_PROPERTIES, ex.getMessage(), ex);
> }
> {code}
> is should be closed after loading.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5363) Create SPENGO-authenticated connection in URLConnectionFactory instead WebHdfsFileSystem
[ https://issues.apache.org/jira/browse/HDFS-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5363: - Status: Patch Available (was: Open) > Create SPENGO-authenticated connection in URLConnectionFactory instead > WebHdfsFileSystem > > > Key: HDFS-5363 > URL: https://issues.apache.org/jira/browse/HDFS-5363 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5363.000.patch > > > Currently the WebHdfsSystem class creates the http connection of urls that > require SPENGO authentication. This patch moves the above logic to > URLConnectionFactory, which is the factory class that supposes to create all > http connection of WebHdfs client. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5363) Create SPENGO-authenticated connection in URLConnectionFactory instead WebHdfsFileSystem
Haohui Mai created HDFS-5363: Summary: Create SPENGO-authenticated connection in URLConnectionFactory instead WebHdfsFileSystem Key: HDFS-5363 URL: https://issues.apache.org/jira/browse/HDFS-5363 Project: Hadoop HDFS Issue Type: Improvement Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5363.000.patch Currently the WebHdfsSystem class creates the http connection of urls that require SPENGO authentication. This patch moves the above logic to URLConnectionFactory, which is the factory class that supposes to create all http connection of WebHdfs client. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5363) Create SPENGO-authenticated connection in URLConnectionFactory instead WebHdfsFileSystem
[ https://issues.apache.org/jira/browse/HDFS-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5363: - Attachment: HDFS-5363.000.patch > Create SPENGO-authenticated connection in URLConnectionFactory instead > WebHdfsFileSystem > > > Key: HDFS-5363 > URL: https://issues.apache.org/jira/browse/HDFS-5363 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5363.000.patch > > > Currently the WebHdfsSystem class creates the http connection of urls that > require SPENGO authentication. This patch moves the above logic to > URLConnectionFactory, which is the factory class that supposes to create all > http connection of WebHdfs client. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-5040: - Assignee: Shinichi Yamashita > Audit log for admin commands/ logging output of all DFS admin commands > -- > > Key: HDFS-5040 > URL: https://issues.apache.org/jira/browse/HDFS-5040 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode >Affects Versions: 3.0.0 >Reporter: Raghu C Doppalapudi >Assignee: Shinichi Yamashita > Attachments: HDFS-5040.patch, HDFS-5040.patch, HDFS-5040.patch > > > enable audit log for all the admin commands/also provide ability to log all > the admin commands in separate log file, at this point all the logging is > displayed on the console. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5352) Server#initLog() doesn't close InputStream in httpfs
[
https://issues.apache.org/jira/browse/HDFS-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-5352:
Resolution: Fixed
Fix Version/s: 2.3.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
+1 for the fix. I've committed this to trunk and branch-2.
> Server#initLog() doesn't close InputStream in httpfs
>
>
> Key: HDFS-5352
> URL: https://issues.apache.org/jira/browse/HDFS-5352
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.3.0
>
> Attachments: hdfs-5352.patch
>
>
> Here is related code snippet in
> hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java:
> {code}
> Properties props = new Properties();
> try {
> InputStream is = getResource(DEFAULT_LOG4J_PROPERTIES);
> props.load(is);
> } catch (IOException ex) {
> throw new ServerException(ServerException.ERROR.S03,
> DEFAULT_LOG4J_PROPERTIES, ex.getMessage(), ex);
> }
> {code}
> is should be closed after loading.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5352) Server#initLog() doesn't close InputStream in httpfs
[
https://issues.apache.org/jira/browse/HDFS-5352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-5352:
Summary: Server#initLog() doesn't close InputStream in httpfs (was:
Server#initLog() doesn't close InputStream)
> Server#initLog() doesn't close InputStream in httpfs
>
>
> Key: HDFS-5352
> URL: https://issues.apache.org/jira/browse/HDFS-5352
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: hdfs-5352.patch
>
>
> Here is related code snippet in
> hadoop-hdfs-project/hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/lib/server/Server.java:
> {code}
> Properties props = new Properties();
> try {
> InputStream is = getResource(DEFAULT_LOG4J_PROPERTIES);
> props.load(is);
> } catch (IOException ex) {
> throw new ServerException(ServerException.ERROR.S03,
> DEFAULT_LOG4J_PROPERTIES, ex.getMessage(), ex);
> }
> {code}
> is should be closed after loading.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5196) Provide more snapshot information in WebUI
[
https://issues.apache.org/jira/browse/HDFS-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794747#comment-13794747
]
Hadoop QA commented on HDFS-5196:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608373/HDFS-5196.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5186//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5186//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5186//console
This message is automatically generated.
> Provide more snapshot information in WebUI
> --
>
> Key: HDFS-5196
> URL: https://issues.apache.org/jira/browse/HDFS-5196
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Affects Versions: 3.0.0
>Reporter: Haohui Mai
>Priority: Minor
> Attachments: HDFS-5196.patch, snapshottable-directoryList.png,
> snapshotteddir.png
>
>
> The WebUI should provide more detailed information about snapshots, such as
> all snapshottable directories and corresponding number of snapshots
> (suggested in HDFS-4096).
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794744#comment-13794744 ] Andrew Wang commented on HDFS-5276: --- Cool, that all works for me. +1 on the new patch; maybe wait a bit in case anyone else wants to ring in, but I think it's fine to commit now if you want. > FileSystem.Statistics got performance issue on multi-thread read/write. > --- > > Key: HDFS-5276 > URL: https://issues.apache.org/jira/browse/HDFS-5276 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Chengxiang Li >Assignee: Colin Patrick McCabe > Attachments: DisableFSReadWriteBytesStat.patch, HDFS-5276.001.patch, > HDFS-5276.002.patch, HDFS-5276.003.patch, HDFSStatisticTest.java, > hdfs-test.PNG, jstack-trace.PNG, TestFileSystemStatistics.java, > ThreadLocalStat.patch > > > FileSystem.Statistics is a singleton variable for each FS scheme, each > read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does > not perform well in multi-threads(let's say more than 30 threads). so it may > cause serious performance issue. during our spark test profile, 32 threads > read data from HDFS, about 70% cpu time is spent on > FileSystem.Statistics.incrementBytesRead(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[ https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5276: --- Attachment: HDFS-5276.003.patch > FileSystem.Statistics got performance issue on multi-thread read/write. > --- > > Key: HDFS-5276 > URL: https://issues.apache.org/jira/browse/HDFS-5276 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Chengxiang Li >Assignee: Colin Patrick McCabe > Attachments: DisableFSReadWriteBytesStat.patch, HDFS-5276.001.patch, > HDFS-5276.002.patch, HDFS-5276.003.patch, HDFSStatisticTest.java, > hdfs-test.PNG, jstack-trace.PNG, TestFileSystemStatistics.java, > ThreadLocalStat.patch > > > FileSystem.Statistics is a singleton variable for each FS scheme, each > read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does > not perform well in multi-threads(let's say more than 30 threads). so it may > cause serious performance issue. during our spark test profile, 32 threads > read data from HDFS, about 70% cpu time is spent on > FileSystem.Statistics.incrementBytesRead(). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[
https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794740#comment-13794740
]
Colin Patrick McCabe commented on HDFS-5276:
thanks for the review, Andrew.
bq. FileSystem#reset is now resetting all the statistics, not just bytes
written and read. This makes sense to me. It looks like the current behavior is
just an oversight from when the additional statistics were added in
HADOOP-6859; maybe Suresh Srinivas as the original author can comment.
Yeah, that's what I think too. Agree that it would be nice to get some
comments on that.
bq. Resetting by taking the current total, #negate-ing, then adding is not
immediately obvious to the reader. Is there a reason we don't just have a
#reset method and reset each StatisticData in turn?
There is a reason. We can't modify the thread-local data... only the threads
themselves can do that, or else there's a race condition. I added a comment
explaining this better.
bq. I'd also prefer just to not reuse the visitor / visitAll [in reset] too,
since it's kind of weird semantically: #total() mutates the total
StatisticsData member, and it depends on #total() being called exactly once at
the end.
I actually thought it was a nice example of code reuse. Iterating over all the
{{StatisticsData}} objects is complex and I would not want to duplicate that
code. I added a comment to {{visitAll}} that describes the fact that
{{aggregate}} (previously called {{total}}) is called exactly once at the end--
hopefully that makes it more obvious.
> FileSystem.Statistics got performance issue on multi-thread read/write.
> ---
>
> Key: HDFS-5276
> URL: https://issues.apache.org/jira/browse/HDFS-5276
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.4-alpha
>Reporter: Chengxiang Li
>Assignee: Colin Patrick McCabe
> Attachments: DisableFSReadWriteBytesStat.patch, HDFS-5276.001.patch,
> HDFS-5276.002.patch, HDFSStatisticTest.java, hdfs-test.PNG, jstack-trace.PNG,
> TestFileSystemStatistics.java, ThreadLocalStat.patch
>
>
> FileSystem.Statistics is a singleton variable for each FS scheme, each
> read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does
> not perform well in multi-threads(let's say more than 30 threads). so it may
> cause serious performance issue. during our spark test profile, 32 threads
> read data from HDFS, about 70% cpu time is spent on
> FileSystem.Statistics.incrementBytesRead().
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5362) Add SnapshotException to terse exception group
Shinichi Yamashita created HDFS-5362: Summary: Add SnapshotException to terse exception group Key: HDFS-5362 URL: https://issues.apache.org/jira/browse/HDFS-5362 Project: Hadoop HDFS Issue Type: Improvement Components: snapshots Affects Versions: 3.0.0 Reporter: Shinichi Yamashita Priority: Minor In trunk, a stack trace of SnapshotException is output NameNode's log via ipc.Server class. The trace of the output method is easy for the message of SnapshotException. So, it should add SnapshotException to terse exception group of NameNodeRpcServer. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HDFS-5359) Allow LightWeightGSet#Iterator to remove elements
[
https://issues.apache.org/jira/browse/HDFS-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved HDFS-5359.
---
Resolution: Fixed
Fix Version/s: HDFS-4949
Committed to branch, thanks Colin.
> Allow LightWeightGSet#Iterator to remove elements
> -
>
> Key: HDFS-5359
> URL: https://issues.apache.org/jira/browse/HDFS-5359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: HDFS-4949
>
> Attachments: HDFS-5359-caching.001.patch
>
>
> Small modification to {{GSet#Iterator}} so that it can remove elements.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5361) Change the unit of StartupProgress 'PercentComplete' to percentage
[ https://issues.apache.org/jira/browse/HDFS-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-5361: Release Note: Change the unit of 'PercentComplete' metric from rate to percentage. Hadoop Flags: Incompatible change Status: Patch Available (was: Open) > Change the unit of StartupProgress 'PercentComplete' to percentage > -- > > Key: HDFS-5361 > URL: https://issues.apache.org/jira/browse/HDFS-5361 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.1.0-beta >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Minor > Labels: metrics, newbie > Attachments: HDFS-5361.patch > > > Now the unit of 'PercentComplete' metrics is rate (maximum is 1.0). It's > confusing for users because its name includes "percent". > The metrics should be multiplied by 100. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5359) Allow LightWeightGSet#Iterator to remove elements
[
https://issues.apache.org/jira/browse/HDFS-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-5359:
--
Summary: Allow LightWeightGSet#Iterator to remove elements (was: allow
LightWeightGSet#Iterator to remove elements)
> Allow LightWeightGSet#Iterator to remove elements
> -
>
> Key: HDFS-5359
> URL: https://issues.apache.org/jira/browse/HDFS-5359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: HDFS-4949
>
> Attachments: HDFS-5359-caching.001.patch
>
>
> Small modification to {{GSet#Iterator}} so that it can remove elements.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5359) allow LightWeightGSet#Iterator to remove elements
[
https://issues.apache.org/jira/browse/HDFS-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794687#comment-13794687
]
Andrew Wang commented on HDFS-5359:
---
Actually, I can't find the submit patch button (maybe because it's an "In
Progress" issue?) so I'm just going to commit this to the branch. We can pull
in any trunk-ready patches later in a separate JIRA.
> allow LightWeightGSet#Iterator to remove elements
> -
>
> Key: HDFS-5359
> URL: https://issues.apache.org/jira/browse/HDFS-5359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-5359-caching.001.patch
>
>
> Small modification to {{GSet#Iterator}} so that it can remove elements.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5361) Change the unit of StartupProgress 'PercentComplete' to percentage
[ https://issues.apache.org/jira/browse/HDFS-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-5361: Attachment: HDFS-5361.patch Attaching a patch. > Change the unit of StartupProgress 'PercentComplete' to percentage > -- > > Key: HDFS-5361 > URL: https://issues.apache.org/jira/browse/HDFS-5361 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.1.0-beta >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Minor > Labels: metrics, newbie > Attachments: HDFS-5361.patch > > > Now the unit of 'PercentComplete' metrics is rate (maximum is 1.0). It's > confusing for users because its name includes "percent". > The metrics should be multiplied by 100. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HDFS-5361) Change the unit of StartupProgress 'PercentComplete' to percentage
[ https://issues.apache.org/jira/browse/HDFS-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reassigned HDFS-5361: --- Assignee: Akira AJISAKA > Change the unit of StartupProgress 'PercentComplete' to percentage > -- > > Key: HDFS-5361 > URL: https://issues.apache.org/jira/browse/HDFS-5361 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.1.0-beta >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Minor > Labels: metrics, newbie > Attachments: HDFS-5361.patch > > > Now the unit of 'PercentComplete' metrics is rate (maximum is 1.0). It's > confusing for users because its name includes "percent". > The metrics should be multiplied by 100. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5359) allow LightWeightGSet#Iterator to remove elements
[
https://issues.apache.org/jira/browse/HDFS-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794680#comment-13794680
]
Andrew Wang commented on HDFS-5359:
---
Cool patch; I had to do some background reading on
ConcurrentModificationException and volatile in HDFS-4522 and friends, but this
patch seems to meet the existing semantics.
+1; I think this is generic enough that it can go in trunk as well as the
branch, so I'm going to mark it PA for jenkins.
> allow LightWeightGSet#Iterator to remove elements
> -
>
> Key: HDFS-5359
> URL: https://issues.apache.org/jira/browse/HDFS-5359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-5359-caching.001.patch
>
>
> Small modification to {{GSet#Iterator}} so that it can remove elements.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5361) Change the unit of StartupProgress 'PercentComplete' to percentage
[ https://issues.apache.org/jira/browse/HDFS-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-5361: Labels: metrics newbie (was: metrics) > Change the unit of StartupProgress 'PercentComplete' to percentage > -- > > Key: HDFS-5361 > URL: https://issues.apache.org/jira/browse/HDFS-5361 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.1.0-beta >Reporter: Akira AJISAKA >Priority: Minor > Labels: metrics, newbie > > Now the unit of 'PercentComplete' metrics is rate (maximum is 1.0). It's > confusing for users because its name includes "percent". > The metrics should be multiplied by 100. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5334) Implement dfshealth.jsp in HTML pages
[
https://issues.apache.org/jira/browse/HDFS-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794649#comment-13794649
]
Hadoop QA commented on HDFS-5334:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608350/HDFS-5334.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5185//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5185//console
This message is automatically generated.
> Implement dfshealth.jsp in HTML pages
> -
>
> Key: HDFS-5334
> URL: https://issues.apache.org/jira/browse/HDFS-5334
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5334.000.patch, HDFS-5334.001.patch,
> HDFS-5334.002.patch, HDFS-5334.003.patch, Screen Shot 2013-10-09 at 10.52.37
> AM.png
>
>
> Reimplement dfshealth.jsp using client-side JavaScript.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5361) Change the unit of StartupProgress 'PercentComplete' to percentage
Akira AJISAKA created HDFS-5361: --- Summary: Change the unit of StartupProgress 'PercentComplete' to percentage Key: HDFS-5361 URL: https://issues.apache.org/jira/browse/HDFS-5361 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.1.0-beta Reporter: Akira AJISAKA Priority: Minor Now the unit of 'PercentComplete' metrics is rate (maximum is 1.0). It's confusing for users because its name includes "percent". The metrics should be multiplied by 100. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5278) Reduce memory consumptions of TestDFSClientRetries
[
https://issues.apache.org/jira/browse/HDFS-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794638#comment-13794638
]
Hadoop QA commented on HDFS-5278:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608351/HDFS-5278.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5184//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5184//console
This message is automatically generated.
> Reduce memory consumptions of TestDFSClientRetries
> --
>
> Key: HDFS-5278
> URL: https://issues.apache.org/jira/browse/HDFS-5278
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-5278.000.patch, HDFS-5278.001.patch
>
>
> TestDFSClientRetries::testDFSClientRetriesOnBusyBlocks() spawns about 50
> threads during the execution, each of which takes more than 6m memory. It
> makes debugging it in eclipse under the default settings difficult since it
> triggers the OutOfMemoryException.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5360) Improvement of usage message of renameSnapshot and deleteSnapshot
Shinichi Yamashita created HDFS-5360:
Summary: Improvement of usage message of renameSnapshot and
deleteSnapshot
Key: HDFS-5360
URL: https://issues.apache.org/jira/browse/HDFS-5360
Project: Hadoop HDFS
Issue Type: Improvement
Components: snapshots
Affects Versions: 3.0.0
Reporter: Shinichi Yamashita
Priority: Minor
When the argument of "hdfs dfs -createSnapshot" comamnd is inappropriate, it is
displayed as follows.
{code}
[hadoop@trunk ~]$ hdfs dfs -createSnapshot
-createSnapshot: is missing.
Usage: hadoop fs [generic options] -createSnapshot
[]
{code}
On the other hands, the commands of "-renameSnapshot" and "-deleteSnapshot" is
displayed as follows. And there are not kind for the user.
{code}
[hadoop@trunk ~]$ hdfs dfs -renameSnapshot
renameSnapshot: args number not 3: 0
[hadoop@trunk ~]$ hdfs dfs -deleteSnapshot
deleteSnapshot: args number not 2: 0
{code}
It changes "-renameSnapshot" and "-deleteSnapshot" to output the message which
is similar to "-createSnapshot".
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5359) allow LightWeightGSet#Iterator to remove elements
[
https://issues.apache.org/jira/browse/HDFS-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-5359:
---
Attachment: HDFS-5359-caching.001.patch
> allow LightWeightGSet#Iterator to remove elements
> -
>
> Key: HDFS-5359
> URL: https://issues.apache.org/jira/browse/HDFS-5359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-5359-caching.001.patch
>
>
> Small modification to {{GSet#Iterator}} so that it can remove elements.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Work started] (HDFS-5359) allow LightWeightGSet#Iterator to remove elements
[
https://issues.apache.org/jira/browse/HDFS-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-5359 started by Colin Patrick McCabe.
> allow LightWeightGSet#Iterator to remove elements
> -
>
> Key: HDFS-5359
> URL: https://issues.apache.org/jira/browse/HDFS-5359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-5359-caching.001.patch
>
>
> Small modification to {{GSet#Iterator}} so that it can remove elements.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5359) allow LightWeightGSet#Iterator to remove elements
[
https://issues.apache.org/jira/browse/HDFS-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-5359:
---
Summary: allow LightWeightGSet#Iterator to remove elements (was: allow
GSet#Iterator to remove elements)
> allow LightWeightGSet#Iterator to remove elements
> -
>
> Key: HDFS-5359
> URL: https://issues.apache.org/jira/browse/HDFS-5359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
>
> Small modification to {{GSet#Iterator}} so that it can remove elements.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5359) allow LightWeightGSet#Iterator to remove elements
[
https://issues.apache.org/jira/browse/HDFS-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794622#comment-13794622
]
Colin Patrick McCabe commented on HDFS-5359:
Note that this doesn't mean supporting concurrent modification and iteration--
just deleting through the iterator, needed for HDFS-5096.
We are not going to support this for the subclass {{LightWeightCache}}, since
that class is a bit more complex and there isn't a use case where we need it
yet.
> allow LightWeightGSet#Iterator to remove elements
> -
>
> Key: HDFS-5359
> URL: https://issues.apache.org/jira/browse/HDFS-5359
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
>
> Small modification to {{GSet#Iterator}} so that it can remove elements.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5359) allow GSet#Iterator to remove elements
Colin Patrick McCabe created HDFS-5359:
--
Summary: allow GSet#Iterator to remove elements
Key: HDFS-5359
URL: https://issues.apache.org/jira/browse/HDFS-5359
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-4949
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Small modification to {{GSet#Iterator}} so that it can remove elements.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5096) Automatically cache new data added to a cached path
[
https://issues.apache.org/jira/browse/HDFS-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794601#comment-13794601
]
Chris Nauroth commented on HDFS-5096:
-
bq. Let me know if you think the comments in the code could be improved.
I think some version of what Andrew just said would do the job nicely, i.e.:
{code}
if (mark != ocblock.getMark()) {
// Mark hasn't been set in this scan, so update replication and mark.
ocblock.setReplicationAndMark(pce.getReplication(), mark);
} else {
// Mark already set in this scan. Set replication to highest value in
// any PathBasedCacheEntry that covers this file.
ocblock.setReplicationAndMark((short)Math.max(
pce.getReplication(), ocblock.getReplication()), mark);
}
{code}
bq. thanks for your patience, both of you
No problem, thanks for your code. :-)
> Automatically cache new data added to a cached path
> ---
>
> Key: HDFS-5096
> URL: https://issues.apache.org/jira/browse/HDFS-5096
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Andrew Wang
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5096-caching.005.patch, HDFS-5096-caching.006.patch
>
>
> For some applications, it's convenient to specify a path to cache, and have
> HDFS automatically cache new data added to the path without sending a new
> caching request or a manual refresh command.
> One example is new data appended to a cached file. It would be nice to
> re-cache a block at the new appended length, and cache new blocks added to
> the file.
> Another example is a cached Hive partition directory, where a user can drop
> new files directly into the partition. It would be nice if these new files
> were cached.
> In both cases, this automatic caching would happen after the file is closed,
> i.e. block replica is finalized.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Resolved] (HDFS-5358) Add replication field to PathBasedCacheDirective
[ https://issues.apache.org/jira/browse/HDFS-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-5358. --- Resolution: Fixed Fix Version/s: HDFS-4949 Hadoop Flags: Reviewed Committed to branch, thanks Colin for the patch and Chris for the review. > Add replication field to PathBasedCacheDirective > > > Key: HDFS-5358 > URL: https://issues.apache.org/jira/browse/HDFS-5358 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: HDFS-4949 > > Attachments: HDFS-5358-caching.001.patch, HDFS-5358-caching.002.patch > > > Add a 'replication' field to PathBasedCacheDirective, so that administrators > can configure how many cached replicas of a block the cluster should try to > maintain. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5358) Add replication field to PathBasedCacheDirective
[ https://issues.apache.org/jira/browse/HDFS-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-5358: -- Summary: Add replication field to PathBasedCacheDirective (was: add 'replication' field to PathBasedCacheDirective) > Add replication field to PathBasedCacheDirective > > > Key: HDFS-5358 > URL: https://issues.apache.org/jira/browse/HDFS-5358 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5358-caching.001.patch, HDFS-5358-caching.002.patch > > > Add a 'replication' field to PathBasedCacheDirective, so that administrators > can configure how many cached replicas of a block the cluster should try to > maintain. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5358) add 'replication' field to PathBasedCacheDirective
[ https://issues.apache.org/jira/browse/HDFS-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794597#comment-13794597 ] Andrew Wang commented on HDFS-5358: --- +1, thanks Colin. Will commit shortly. > add 'replication' field to PathBasedCacheDirective > -- > > Key: HDFS-5358 > URL: https://issues.apache.org/jira/browse/HDFS-5358 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5358-caching.001.patch, HDFS-5358-caching.002.patch > > > Add a 'replication' field to PathBasedCacheDirective, so that administrators > can configure how many cached replicas of a block the cluster should try to > maintain. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5196) Provide more snapshot information in WebUI
[ https://issues.apache.org/jira/browse/HDFS-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita updated HDFS-5196: - Target Version/s: 3.0.0 Affects Version/s: 3.0.0 Status: Patch Available (was: Open) > Provide more snapshot information in WebUI > -- > > Key: HDFS-5196 > URL: https://issues.apache.org/jira/browse/HDFS-5196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Affects Versions: 3.0.0 >Reporter: Haohui Mai >Priority: Minor > Attachments: HDFS-5196.patch, snapshottable-directoryList.png, > snapshotteddir.png > > > The WebUI should provide more detailed information about snapshots, such as > all snapshottable directories and corresponding number of snapshots > (suggested in HDFS-4096). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5196) Provide more snapshot information in WebUI
[ https://issues.apache.org/jira/browse/HDFS-5196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita updated HDFS-5196: - Attachment: HDFS-5196.patch I attach a patch file about two images of the former comment. > Provide more snapshot information in WebUI > -- > > Key: HDFS-5196 > URL: https://issues.apache.org/jira/browse/HDFS-5196 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Haohui Mai >Priority: Minor > Attachments: HDFS-5196.patch, snapshottable-directoryList.png, > snapshotteddir.png > > > The WebUI should provide more detailed information about snapshots, such as > all snapshottable directories and corresponding number of snapshots > (suggested in HDFS-4096). -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5358) add 'replication' field to PathBasedCacheDirective
[ https://issues.apache.org/jira/browse/HDFS-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5358: --- Attachment: HDFS-5358-caching.002.patch > add 'replication' field to PathBasedCacheDirective > -- > > Key: HDFS-5358 > URL: https://issues.apache.org/jira/browse/HDFS-5358 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5358-caching.001.patch, HDFS-5358-caching.002.patch > > > Add a 'replication' field to PathBasedCacheDirective, so that administrators > can configure how many cached replicas of a block the cluster should try to > maintain. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5096) Automatically cache new data added to a cached path
[ https://issues.apache.org/jira/browse/HDFS-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794590#comment-13794590 ] Colin Patrick McCabe commented on HDFS-5096: and yeah, taking the FSN lock while initializing the CacheManager was deliberate, to avoid having to get rid of some "assert lock is held" code. > Automatically cache new data added to a cached path > --- > > Key: HDFS-5096 > URL: https://issues.apache.org/jira/browse/HDFS-5096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Andrew Wang >Assignee: Colin Patrick McCabe > Attachments: HDFS-5096-caching.005.patch, HDFS-5096-caching.006.patch > > > For some applications, it's convenient to specify a path to cache, and have > HDFS automatically cache new data added to the path without sending a new > caching request or a manual refresh command. > One example is new data appended to a cached file. It would be nice to > re-cache a block at the new appended length, and cache new blocks added to > the file. > Another example is a cached Hive partition directory, where a user can drop > new files directly into the partition. It would be nice if these new files > were cached. > In both cases, this automatic caching would happen after the file is closed, > i.e. block replica is finalized. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5096) Automatically cache new data added to a cached path
[ https://issues.apache.org/jira/browse/HDFS-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794588#comment-13794588 ] Colin Patrick McCabe commented on HDFS-5096: bq. Shall we remove INodeFile#cacheReplication in this patch? I believe it's unused now. Good point. bq. Also, can we please document dfs.namenode.path.based.cache.refresh.interval.ms in hdfs-default.xml? added. bq. CacheReplicationMonitor#rescanFile I agree with Andrew's comments here... sorry for not replying earlier but I've been focusing on breaking out pieces into HDFS-5358, HDFS-5348, HDFS-5349, and the other JIRAs. Let me know if you think the comments in the code could be improved. I still haven't replied to all of andrew's comments either... thanks for your patience, both of you :) > Automatically cache new data added to a cached path > --- > > Key: HDFS-5096 > URL: https://issues.apache.org/jira/browse/HDFS-5096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Andrew Wang >Assignee: Colin Patrick McCabe > Attachments: HDFS-5096-caching.005.patch, HDFS-5096-caching.006.patch > > > For some applications, it's convenient to specify a path to cache, and have > HDFS automatically cache new data added to the path without sending a new > caching request or a manual refresh command. > One example is new data appended to a cached file. It would be nice to > re-cache a block at the new appended length, and cache new blocks added to > the file. > Another example is a cached Hive partition directory, where a user can drop > new files directly into the partition. It would be nice if these new files > were cached. > In both cases, this automatic caching would happen after the file is closed, > i.e. block replica is finalized. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5324) Make Namespace implementation pluggable in the namenode
[ https://issues.apache.org/jira/browse/HDFS-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794584#comment-13794584 ] Milind Bhandarkar commented on HDFS-5324: - Colin, Yes, I agree that the big lock covering the entire name system needs to be moved into the NS implementation, if needed, and we need to exclude BlockManager from using this same lock. This is precisely why an work-in-progress drop of this code was made available, so that such insightful comments are made and incorporated into the design. We have some ideas about how to do this, and I will post code snippets with that even before the patch is uploaded. Regarding the API, I suggest that it should be InterfaceAudience Private, and the class itself should be public. So that hdfs developers can modify it anytime they want, and those who want to keep their namespace implementation separate from hadoop-hdfs project (e.g. hbase-based impl) can still use it. > Make Namespace implementation pluggable in the namenode > --- > > Key: HDFS-5324 > URL: https://issues.apache.org/jira/browse/HDFS-5324 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.1.1-beta > Environment: All >Reporter: Milind Bhandarkar >Assignee: Milind Bhandarkar > Fix For: 3.0.0 > > Attachments: AbstractNamesystem.java > > > For the last couple of months, we have been working on making Namespace > implementation in the namenode pluggable. We have demonstrated that it can > be done without major surgery on the namenode, and does not have noticeable > performance impact. We would like to contribute it back to Apache HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5358) add 'replication' field to PathBasedCacheDirective
[
https://issues.apache.org/jira/browse/HDFS-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794567#comment-13794567
]
Colin Patrick McCabe commented on HDFS-5358:
bq. Can we use uint32 for replication in the protocol? This would further
prevent a negative value, and also it would be consistent with the replication
field in other protocol messages. (See example below.)
OK. I will change it to use this for consistency.
bq. Higher-level question, what do you think about a "cache all" replication
factor...
Let's worry about having a "cache all" value later. That really deserves its
own JIRA. Probably we could use {{Short.MIN_VALUE}} or 0 for that. There are
no shortage of reserved values, after all-- we send 32 bits over the wire but
only have 15 bits in a positive java short.
bq. Any reason why the edit log loader is no longer using the unprotected
functions? It should be safe to skip the checks, which makes edit log replaying
faster. If you do want to go with this, we should get rid of the unprotected
functions where possible / update the javadoc.
That must have leaked in when I ported this from 5096. Will fix.
> add 'replication' field to PathBasedCacheDirective
> --
>
> Key: HDFS-5358
> URL: https://issues.apache.org/jira/browse/HDFS-5358
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5358-caching.001.patch
>
>
> Add a 'replication' field to PathBasedCacheDirective, so that administrators
> can configure how many cached replicas of a block the cluster should try to
> maintain.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5096) Automatically cache new data added to a cached path
[
https://issues.apache.org/jira/browse/HDFS-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794555#comment-13794555
]
Chris Nauroth commented on HDFS-5096:
-
Thanks, Colin and Andrew. I agree that the logic looks correct on
pending-cached/pending-uncached and the mark check. It was just a matter of
comments.
Shall we remove {{INodeFile#cacheReplication}} in this patch? I believe it's
unused now.
> Automatically cache new data added to a cached path
> ---
>
> Key: HDFS-5096
> URL: https://issues.apache.org/jira/browse/HDFS-5096
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Andrew Wang
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5096-caching.005.patch, HDFS-5096-caching.006.patch
>
>
> For some applications, it's convenient to specify a path to cache, and have
> HDFS automatically cache new data added to the path without sending a new
> caching request or a manual refresh command.
> One example is new data appended to a cached file. It would be nice to
> re-cache a block at the new appended length, and cache new blocks added to
> the file.
> Another example is a cached Hive partition directory, where a user can drop
> new files directly into the partition. It would be nice if these new files
> were cached.
> In both cases, this automatic caching would happen after the file is closed,
> i.e. block replica is finalized.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5096) Automatically cache new data added to a cached path
[
https://issues.apache.org/jira/browse/HDFS-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794556#comment-13794556
]
Chris Nauroth commented on HDFS-5096:
-
Also, can we please document
{{dfs.namenode.path.based.cache.refresh.interval.ms}} in hdfs-default.xml?
> Automatically cache new data added to a cached path
> ---
>
> Key: HDFS-5096
> URL: https://issues.apache.org/jira/browse/HDFS-5096
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Andrew Wang
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5096-caching.005.patch, HDFS-5096-caching.006.patch
>
>
> For some applications, it's convenient to specify a path to cache, and have
> HDFS automatically cache new data added to the path without sending a new
> caching request or a manual refresh command.
> One example is new data appended to a cached file. It would be nice to
> re-cache a block at the new appended length, and cache new blocks added to
> the file.
> Another example is a cached Hive partition directory, where a user can drop
> new files directly into the partition. It would be nice if these new files
> were cached.
> In both cases, this automatic caching would happen after the file is closed,
> i.e. block replica is finalized.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5349) DNA_CACHE and DNA_UNCACHE should be by blockId only
[ https://issues.apache.org/jira/browse/HDFS-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794547#comment-13794547 ] Colin Patrick McCabe commented on HDFS-5349: bq. I think this is getting cleaned up in HDFS-5096, but In DatanodeManager, it'll be nice to make forming the list of blockIds into a method. Yeah, this is cleaned up in 5096 bq. In BPOfferService, might be more clear to have a separate switch for BlockIdCommand. Let's do that in the future to keep the diff down thanks > DNA_CACHE and DNA_UNCACHE should be by blockId only > > > Key: HDFS-5349 > URL: https://issues.apache.org/jira/browse/HDFS-5349 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: HDFS-4949 > > Attachments: HDFS-5349-caching.001.patch, HDFS-5349-caching.002.patch > > > DNA_CACHE and DNA_UNCACHE should be by blockId only. We don't need length > and genstamp to know what the NN asked us to cache. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HDFS-5349) DNA_CACHE and DNA_UNCACHE should be by blockId only
[ https://issues.apache.org/jira/browse/HDFS-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HDFS-5349. Resolution: Fixed Fix Version/s: HDFS-4949 > DNA_CACHE and DNA_UNCACHE should be by blockId only > > > Key: HDFS-5349 > URL: https://issues.apache.org/jira/browse/HDFS-5349 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: HDFS-4949 > > Attachments: HDFS-5349-caching.001.patch, HDFS-5349-caching.002.patch > > > DNA_CACHE and DNA_UNCACHE should be by blockId only. We don't need length > and genstamp to know what the NN asked us to cache. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5096) Automatically cache new data added to a cached path
[
https://issues.apache.org/jira/browse/HDFS-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794540#comment-13794540
]
Andrew Wang commented on HDFS-5096:
---
Hey Chris, I can field a few of these for Colin:
bq. Something seems off about the logic...
I stumbled on this too, so the comment should be improved. I think the logic is
correct though:
* If the # of cached replicas is geq than the desired cache replication factor,
then clear any pending cache operations
* If the number cached is less than the desired cache replication factor, then
clear pending uncache operations
Maybe saying {{numCached >= neededRepl}} and {{numCached < neededRepl}} would
be clearer, as well as renaming {{neededReplication}} to {{desiredCached}} or
{{cacheReplFactor}} and rewording the javadoc.
bq. CRMon#rescanFile...
If the mark hasn't been set, we use the repl PCE. Else if it's already been
visited during this rescan (as evidenced via the mark already being set), we
want it to be the max of previous PCE repl factors and this PCE. This handles
duplicate PCEs for the same file, and could definitely use a comment.
bq. NameNode: Is this HA change meant for this patch...
I think this is so we can assert the write lock in {{CacheManager#activate()}}.
I think holding the write lock here makes sense in general, so it could be
punted to trunk after this to reduce the branch diff.
> Automatically cache new data added to a cached path
> ---
>
> Key: HDFS-5096
> URL: https://issues.apache.org/jira/browse/HDFS-5096
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Andrew Wang
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5096-caching.005.patch, HDFS-5096-caching.006.patch
>
>
> For some applications, it's convenient to specify a path to cache, and have
> HDFS automatically cache new data added to the path without sending a new
> caching request or a manual refresh command.
> One example is new data appended to a cached file. It would be nice to
> re-cache a block at the new appended length, and cache new blocks added to
> the file.
> Another example is a cached Hive partition directory, where a user can drop
> new files directly into the partition. It would be nice if these new files
> were cached.
> In both cases, this automatic caching would happen after the file is closed,
> i.e. block replica is finalized.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5096) Automatically cache new data added to a cached path
[
https://issues.apache.org/jira/browse/HDFS-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794539#comment-13794539
]
Colin Patrick McCabe commented on HDFS-5096:
bq. Looks like the TODO: support loading and storing snuck back in during a
rebase
Fixed.
bq. I'd lower the cachedBlocks GSet size to something like 1 or 2%. This makes
sense considering the blocksMap is set to 50% of heap, and there should be at
least an order of magnitude fewer cached blocks.
It's actually at 0.25% now. {{LightWeightGSet#computeCapacity}} does things in
percentages.
bq. Is adding the iterator to LightWeightCache necessary? Seems like that
should go on trunk instead, and there are no callers.
It's necessary to protect callers from getting undefined behavior when they
call {{Iterator#remove}}. It's just a simple wrapper, not a lot of code.
bq. Seems like a generally useful class, could we put this in common?
Sure.
bq. Rather than randomly deciding to remove via iterator or pseudo-randomly,
let's just test each case definitively.
I use Random with a pre-defined seed, so it's deterministic.
bq. [CachedBlock] should check for setting an invalid replication factor,
especially since we're re-using the top bit for the mark.
Added assert.
bq. I still think we should code share somehow with BlockInfo, having all this
triplets logic twice sucks. This could go into a trunk refactor patch.
I don't think it's really possible. BlockInfo already inherits from Block, and
Java doesn't support multiple inheritance. We can't use composition here
because the overhead of having another object inside each Block would be too
high. Perhaps we could refactor them both to support the same interface, but
it would be a big job.
bq. I think we should be taking the write lock, not the read lock in #rescan,
since in rescanCachedBlockMap we're modifying things, e.g. removeElement,
addNewPendingCached.
Yeah, I suppose by writing to cachedBlocks, we might be conflicting with other
threads that also hold the read lock. such as threads getting information
about cached replicas. I'll change it to take the write lock for now.
bq. Javadoc incorrect on #rescanCachedBlockMap, those parameters don't exist
Fixed.
bq. Rename neededReplication to neededCached for uniformity
OK.
bq. Comment for neededReplicaion >= numCached needs to be updated (it's just
copy pasted from the above)
OK.
bq. The neededCached case is passing in pendingUncached instead of pendingCached
fixed, thanks.
bq. I think it's okay to populate the pending queues and do all the tracking
even in standby mode, since it'd be good to take over these duties immediately
on failover. We just shouldn't send any datanode operations.
Partly, I want to avoid having this thread run during cases like when
formatting the namespace, and running it only when active is a good way to do
that. Also, keep in mind that activating the CacheManager will kick off an
immediate rescan, so we won't have long to wait before we start sending off
cache directives after becoming active.
bq. processCacheReport, shouldReply is unused.
I got rid of all that; thanks.
bq. This merged class no longer uses the ReportProcessor class we refactored
out from BlockManager. I like ReportProcessor, but it's of dubious utility now.
We should either punt this change out to trunk, or revert it to keep our diff
down.
Good point. I'll try to sync us up with trunk here.
bq. Is there some way to enforce that a block is never present on multiple of a
datanode's CachedBlockLists?
A block can absolutely be on multiple of a datanode's CachedBlockLists. For
example, it can be on the 'cached' list and also on the 'pendingUncached' list,
if we know about it but want to get rid of it.
bq. This would be clearer without prevCachedBlock
I added a comment clarifying how this works.
I'm going to add these comments now so that I don't lose them if I close the
window. Still have a few to address, I realize.
> Automatically cache new data added to a cached path
> ---
>
> Key: HDFS-5096
> URL: https://issues.apache.org/jira/browse/HDFS-5096
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Andrew Wang
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5096-caching.005.patch, HDFS-5096-caching.006.patch
>
>
> For some applications, it's convenient to specify a path to cache, and have
> HDFS automatically cache new data added to the path without sending a new
> caching request or a manual refresh command.
> One example is new data appended to a cached file. It would be nice to
> re-cache a block at the new appended length, and cache new blocks added to
> the file.
> Another example is a cached Hive partition direct
[jira] [Commented] (HDFS-5357) TestFileSystemAccessService failures in JDK7
[ https://issues.apache.org/jira/browse/HDFS-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794519#comment-13794519 ] Jonathan Eagles commented on HDFS-5357: --- +1. Thanks, Rob. This patch is only for branch 0.23, so ignoring the Hadoop QA results. This tests now runs successfully consistently with this change. > TestFileSystemAccessService failures in JDK7 > > > Key: HDFS-5357 > URL: https://issues.apache.org/jira/browse/HDFS-5357 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.9 >Reporter: Robert Parker >Assignee: Robert Parker > Attachments: HDFS-5357v1.patch > > > junit.framework.AssertionFailedError: Expected Exception: ServiceException > got: ExceptionInInitializerError > at junit.framework.Assert.fail(Assert.java:47) > at > org.apache.hadoop.test.TestExceptionHelper$1.evaluate(TestExceptionHelper.java:56) > at > org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) > at org.junit.runners.ParentRunner.run(ParentRunner.java:236) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5324) Make Namespace implementation pluggable in the namenode
[
https://issues.apache.org/jira/browse/HDFS-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794517#comment-13794517
]
Colin Patrick McCabe commented on HDFS-5324:
As you know, we currently have a reader/writer lock which protects almost
everything in {{FSNamesystem}}. This lock is used in a lot of places, such as
in the {{BlockManager}}, not just inside the {{FSNamesystem}} class itself.
Having a "big lock" such as this is fundamentally incompatible with having a
paged namespace. The problem is, if you take the lock, and then do something
that requires you to go out to disk, other threads could hang for seconds or
minutes at a time. Disk I/O is unpredictable and often takes a while.
Currently, we never have to worry about this, since everything in the
{{FSNamesystem}} is held in memory, and memory can be accessed relatively
quickly.
If you can come up with patches to implement fine-grained locking, then we
could start thinking about code sharing. But until that point, I don't see how
it's possible to share much code here.
bq. As I have replied to Sanjay, I am not proposing a stable API within HDFS
for implementing Namespaces
If that's true, then the API should be private to HDFS, and your alternate FSN
implementation should be in the hadoop-hdfs project, right?
> Make Namespace implementation pluggable in the namenode
> ---
>
> Key: HDFS-5324
> URL: https://issues.apache.org/jira/browse/HDFS-5324
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.1.1-beta
> Environment: All
>Reporter: Milind Bhandarkar
>Assignee: Milind Bhandarkar
> Fix For: 3.0.0
>
> Attachments: AbstractNamesystem.java
>
>
> For the last couple of months, we have been working on making Namespace
> implementation in the namenode pluggable. We have demonstrated that it can
> be done without major surgery on the namenode, and does not have noticeable
> performance impact. We would like to contribute it back to Apache HDFS.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5278) Reduce memory consumptions of TestDFSClientRetries
[ https://issues.apache.org/jira/browse/HDFS-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5278: - Attachment: HDFS-5278.001.patch > Reduce memory consumptions of TestDFSClientRetries > -- > > Key: HDFS-5278 > URL: https://issues.apache.org/jira/browse/HDFS-5278 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5278.000.patch, HDFS-5278.001.patch > > > TestDFSClientRetries::testDFSClientRetriesOnBusyBlocks() spawns about 50 > threads during the execution, each of which takes more than 6m memory. It > makes debugging it in eclipse under the default settings difficult since it > triggers the OutOfMemoryException. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5334) Implement dfshealth.jsp in HTML pages
[ https://issues.apache.org/jira/browse/HDFS-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5334: - Attachment: HDFS-5334.003.patch Based on HDFS-5342. > Implement dfshealth.jsp in HTML pages > - > > Key: HDFS-5334 > URL: https://issues.apache.org/jira/browse/HDFS-5334 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5334.000.patch, HDFS-5334.001.patch, > HDFS-5334.002.patch, HDFS-5334.003.patch, Screen Shot 2013-10-09 at 10.52.37 > AM.png > > > Reimplement dfshealth.jsp using client-side JavaScript. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5358) add 'replication' field to PathBasedCacheDirective
[
https://issues.apache.org/jira/browse/HDFS-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794503#comment-13794503
]
Andrew Wang commented on HDFS-5358:
---
Thanks for splitting this out from HDFS-5096, nice to get that logical
separation as well.
* Higher-level question, what do you think about a "cache all" replication
factor that just uses the underlying file's replication factor? I think 1 is a
good default, but I could imagine blowing up the replication factor of a small
often-used table and also wanting it to be cached.
* Turning {{0}} into a {{public static final short MIN_REPLICATION = 1}} or
(better) a {{validate()}} check would make it easier if we later decide to
reserve some numbers for special values like "cache all"
* Any reason why the edit log loader is no longer using the unprotected
functions? It should be safe to skip the checks, which makes edit log replaying
faster. If you do want to go with this, we should get rid of the unprotected
functions where possible / update the javadoc.
> add 'replication' field to PathBasedCacheDirective
> --
>
> Key: HDFS-5358
> URL: https://issues.apache.org/jira/browse/HDFS-5358
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5358-caching.001.patch
>
>
> Add a 'replication' field to PathBasedCacheDirective, so that administrators
> can configure how many cached replicas of a block the cluster should try to
> maintain.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5096) Automatically cache new data added to a cached path
[
https://issues.apache.org/jira/browse/HDFS-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794501#comment-13794501
]
Chris Nauroth commented on HDFS-5096:
-
A couple of quick comments looking at version 6 of the patch:
{{CacheReplicationMonitor#rescanCachedBlockMap}}: Something seems off about the
logic for manipulating pending-cached and pending-uncached. Is it just that
the comments are wrong and I'm getting confused?
{code}
if (neededReplication <= numCached) {
// If we have all the replicas we need, or too few, drop all
// pending cached.
for (DatanodeDescriptor datanode : pendingCached) {
datanode.getPendingCached().removeElement(cblock);
}
}
if (neededReplication >= numCached) {
// If we have all the replicas we need, or too many, drop all
// pending cached.
for (DatanodeDescriptor datanode : pendingUncached) {
datanode.getPendingUncached().removeElement(cblock);
}
}
{code}
{{CacheReplicationMonitor#rescanFile}}: Can you explain the logic around mark
in this method? I understand the mark logic in {{rescanCachedBlockMap}}, but I
didn't follow it here.
{code}
if (mark != ocblock.getMark()) {
ocblock.setReplicationAndMark(pce.getReplication(), mark);
} else {
ocblock.setReplicationAndMark((short)Math.max(
pce.getReplication(), ocblock.getReplication()), mark);
}
{code}
{{NameNode}}: Is this HA change meant for this patch, or is it meant to be its
own patch that can go to trunk?
Several tests are commented out in this version of the patch so that they
aren't running.
> Automatically cache new data added to a cached path
> ---
>
> Key: HDFS-5096
> URL: https://issues.apache.org/jira/browse/HDFS-5096
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Andrew Wang
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5096-caching.005.patch, HDFS-5096-caching.006.patch
>
>
> For some applications, it's convenient to specify a path to cache, and have
> HDFS automatically cache new data added to the path without sending a new
> caching request or a manual refresh command.
> One example is new data appended to a cached file. It would be nice to
> re-cache a block at the new appended length, and cache new blocks added to
> the file.
> Another example is a cached Hive partition directory, where a user can drop
> new files directly into the partition. It would be nice if these new files
> were cached.
> In both cases, this automatic caching would happen after the file is closed,
> i.e. block replica is finalized.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5342) Provide more information in the FSNamesystem JMX interfaces
[ https://issues.apache.org/jira/browse/HDFS-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794485#comment-13794485 ] Hudson commented on HDFS-5342: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4601 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4601/]) HDFS-5342. Provide more information in the FSNamesystem JMX interfaces. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1532090) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/StartupProgressServlet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/FSNamesystemMBean.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartupProgressServlet.java > Provide more information in the FSNamesystem JMX interfaces > --- > > Key: HDFS-5342 > URL: https://issues.apache.org/jira/browse/HDFS-5342 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.3.0 > > Attachments: HDFS-5342.000.patch, HDFS-5342.001.patch, > HDFS-5342.002.patch > > > Today the JMX interface in HDFS only provides a subset of information listed > in the WebUI. This jira proposes to augment the JMX interface so that it > provides as much information as the Web UI today. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5358) add 'replication' field to PathBasedCacheDirective
[
https://issues.apache.org/jira/browse/HDFS-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794484#comment-13794484
]
Chris Nauroth commented on HDFS-5358:
-
Can we use uint32 for replication in the protocol? This would further prevent
a negative value, and also it would be consistent with the replication field in
other protocol messages. (See example below.)
Aside from that, the patch looks good.
{code}
message SetReplicationRequestProto {
required string src = 1;
required uint32 replication = 2; // Short: Only 16 bits used
}
{code}
> add 'replication' field to PathBasedCacheDirective
> --
>
> Key: HDFS-5358
> URL: https://issues.apache.org/jira/browse/HDFS-5358
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5358-caching.001.patch
>
>
> Add a 'replication' field to PathBasedCacheDirective, so that administrators
> can configure how many cached replicas of a block the cluster should try to
> maintain.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5342) Provide more information in the FSNamesystem JMX interfaces
[ https://issues.apache.org/jira/browse/HDFS-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5342: Resolution: Fixed Fix Version/s: 2.3.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk and branch-2. > Provide more information in the FSNamesystem JMX interfaces > --- > > Key: HDFS-5342 > URL: https://issues.apache.org/jira/browse/HDFS-5342 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 2.3.0 > > Attachments: HDFS-5342.000.patch, HDFS-5342.001.patch, > HDFS-5342.002.patch > > > Today the JMX interface in HDFS only provides a subset of information listed > in the WebUI. This jira proposes to augment the JMX interface so that it > provides as much information as the Web UI today. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5342) Provide more information in the FSNamesystem JMX interfaces
[ https://issues.apache.org/jira/browse/HDFS-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5342: Summary: Provide more information in the FSNamesystem JMX interfaces (was: Provide more information in the JMX interfaces) > Provide more information in the FSNamesystem JMX interfaces > --- > > Key: HDFS-5342 > URL: https://issues.apache.org/jira/browse/HDFS-5342 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5342.000.patch, HDFS-5342.001.patch, > HDFS-5342.002.patch > > > Today the JMX interface in HDFS only provides a subset of information listed > in the WebUI. This jira proposes to augment the JMX interface so that it > provides as much information as the Web UI today. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5342) Provide more information in the JMX interfaces
[ https://issues.apache.org/jira/browse/HDFS-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794463#comment-13794463 ] Jing Zhao commented on HDFS-5342: - Thanks for the update Haohui! The patch looks good to me. I will commit it shortly. > Provide more information in the JMX interfaces > -- > > Key: HDFS-5342 > URL: https://issues.apache.org/jira/browse/HDFS-5342 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5342.000.patch, HDFS-5342.001.patch, > HDFS-5342.002.patch > > > Today the JMX interface in HDFS only provides a subset of information listed > in the WebUI. This jira proposes to augment the JMX interface so that it > provides as much information as the Web UI today. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5349) DNA_CACHE and DNA_UNCACHE should be by blockId only
[
https://issues.apache.org/jira/browse/HDFS-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794457#comment-13794457
]
Andrew Wang commented on HDFS-5349:
---
Great, looks good. +1, up to you on the following:
* I think this is getting cleaned up in HDFS-5096, but In DatanodeManager,
it'll be nice to make forming the list of blockIds into a method.
* In {{BPOfferService}}, might be more clear to have a separate switch for
{{BlockIdCommand}}.
Thanks for going through and updating all the javadoc too.
> DNA_CACHE and DNA_UNCACHE should be by blockId only
>
>
> Key: HDFS-5349
> URL: https://issues.apache.org/jira/browse/HDFS-5349
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Affects Versions: HDFS-4949
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-5349-caching.001.patch, HDFS-5349-caching.002.patch
>
>
> DNA_CACHE and DNA_UNCACHE should be by blockId only. We don't need length
> and genstamp to know what the NN asked us to cache.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5356) MiniDFSCluster shoud close all open FileSystems when shutdown()
[
https://issues.apache.org/jira/browse/HDFS-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794443#comment-13794443
]
Hadoop QA commented on HDFS-5356:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608314/HDFS-5356.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
org.apache.hadoop.hdfs.server.namenode.ha.TestHAAppend
org.apache.hadoop.hdfs.server.namenode.TestINodeFile
org.apache.hadoop.hdfs.server.namenode.TestNameNodeRecovery
org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits
org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
org.apache.hadoop.hdfs.TestDFSClientFailover
org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM
org.apache.hadoop.hdfs.web.TestWebHDFSForHA
org.apache.hadoop.hdfs.TestFileCreation
org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandby
org.apache.hadoop.hdfs.server.namenode.ha.TestDFSZKFailoverController
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5182//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5182//console
This message is automatically generated.
> MiniDFSCluster shoud close all open FileSystems when shutdown()
> ---
>
> Key: HDFS-5356
> URL: https://issues.apache.org/jira/browse/HDFS-5356
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: haosdent
>Priority: Critical
> Attachments: HDFS-5356.patch
>
>
> After add some metrics functions to DFSClient, I found that some unit tests
> relates to metrics are failed. Because MiniDFSCluster are never close open
> FileSystems, DFSClients are alive after MiniDFSCluster shutdown(). The
> metrics of DFSClients in DefaultMetricsSystem are still exist and this make
> other unit tests failed.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5096) Automatically cache new data added to a cached path
[ https://issues.apache.org/jira/browse/HDFS-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794408#comment-13794408 ] Colin Patrick McCabe commented on HDFS-5096: I split off the change adding replication into HDFS-5358 > Automatically cache new data added to a cached path > --- > > Key: HDFS-5096 > URL: https://issues.apache.org/jira/browse/HDFS-5096 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Andrew Wang >Assignee: Colin Patrick McCabe > Attachments: HDFS-5096-caching.005.patch, HDFS-5096-caching.006.patch > > > For some applications, it's convenient to specify a path to cache, and have > HDFS automatically cache new data added to the path without sending a new > caching request or a manual refresh command. > One example is new data appended to a cached file. It would be nice to > re-cache a block at the new appended length, and cache new blocks added to > the file. > Another example is a cached Hive partition directory, where a user can drop > new files directly into the partition. It would be nice if these new files > were cached. > In both cases, this automatic caching would happen after the file is closed, > i.e. block replica is finalized. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5358) add 'replication' field to PathBasedCacheDirective
[ https://issues.apache.org/jira/browse/HDFS-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5358: --- Attachment: HDFS-5358-caching.001.patch > add 'replication' field to PathBasedCacheDirective > -- > > Key: HDFS-5358 > URL: https://issues.apache.org/jira/browse/HDFS-5358 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5358-caching.001.patch > > > Add a 'replication' field to PathBasedCacheDirective, so that administrators > can configure how many cached replicas of a block the cluster should try to > maintain. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Work started] (HDFS-5358) add 'replication' field to PathBasedCacheDirective
[ https://issues.apache.org/jira/browse/HDFS-5358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-5358 started by Colin Patrick McCabe. > add 'replication' field to PathBasedCacheDirective > -- > > Key: HDFS-5358 > URL: https://issues.apache.org/jira/browse/HDFS-5358 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5358-caching.001.patch > > > Add a 'replication' field to PathBasedCacheDirective, so that administrators > can configure how many cached replicas of a block the cluster should try to > maintain. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HDFS-5346) Replication queues should not be initialized in the middle of IBR processing.
[ https://issues.apache.org/jira/browse/HDFS-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash reassigned HDFS-5346: -- Assignee: Ravi Prakash > Replication queues should not be initialized in the middle of IBR processing. > - > > Key: HDFS-5346 > URL: https://issues.apache.org/jira/browse/HDFS-5346 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, performance >Affects Versions: 0.23.9, 2.3.0 >Reporter: Kihwal Lee >Assignee: Ravi Prakash > Fix For: 2.3.0, 0.23.10 > > > When initial block reports are being processed, checkMode() is called from > incrementSafeBlockCount(). This causes the replication queues to be > initialized in the middle of processing a block report in the IBR processing > mode. If there are many block reports waiting to be processed, > SafeModeMonitor won't be able to make name node leave the safe mode soon. It > appears that the block report processing speed degrades considerably during > this time. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5357) TestFileSystemAccessService failures in JDK7
[
https://issues.apache.org/jira/browse/HDFS-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794406#comment-13794406
]
Hadoop QA commented on HDFS-5357:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608332/HDFS-5357v1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5183//console
This message is automatically generated.
> TestFileSystemAccessService failures in JDK7
>
>
> Key: HDFS-5357
> URL: https://issues.apache.org/jira/browse/HDFS-5357
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.9
>Reporter: Robert Parker
>Assignee: Robert Parker
> Attachments: HDFS-5357v1.patch
>
>
> junit.framework.AssertionFailedError: Expected Exception: ServiceException
> got: ExceptionInInitializerError
> at junit.framework.Assert.fail(Assert.java:47)
> at
> org.apache.hadoop.test.TestExceptionHelper$1.evaluate(TestExceptionHelper.java:56)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5357) TestFileSystemAccessService failures in JDK7
[ https://issues.apache.org/jira/browse/HDFS-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated HDFS-5357: Status: Patch Available (was: Open) > TestFileSystemAccessService failures in JDK7 > > > Key: HDFS-5357 > URL: https://issues.apache.org/jira/browse/HDFS-5357 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.9 >Reporter: Robert Parker >Assignee: Robert Parker > Attachments: HDFS-5357v1.patch > > > junit.framework.AssertionFailedError: Expected Exception: ServiceException > got: ExceptionInInitializerError > at junit.framework.Assert.fail(Assert.java:47) > at > org.apache.hadoop.test.TestExceptionHelper$1.evaluate(TestExceptionHelper.java:56) > at > org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) > at org.junit.runners.ParentRunner.run(ParentRunner.java:236) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5357) TestFileSystemAccessService failures in JDK7
[ https://issues.apache.org/jira/browse/HDFS-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated HDFS-5357: Attachment: HDFS-5357v1.patch > TestFileSystemAccessService failures in JDK7 > > > Key: HDFS-5357 > URL: https://issues.apache.org/jira/browse/HDFS-5357 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.9 >Reporter: Robert Parker >Assignee: Robert Parker > Attachments: HDFS-5357v1.patch > > > junit.framework.AssertionFailedError: Expected Exception: ServiceException > got: ExceptionInInitializerError > at junit.framework.Assert.fail(Assert.java:47) > at > org.apache.hadoop.test.TestExceptionHelper$1.evaluate(TestExceptionHelper.java:56) > at > org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) > at org.junit.runners.ParentRunner.run(ParentRunner.java:236) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HDFS-5357) TestFileSystemAccessService failures in JDK7
[ https://issues.apache.org/jira/browse/HDFS-5357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker reassigned HDFS-5357: --- Assignee: Robert Parker > TestFileSystemAccessService failures in JDK7 > > > Key: HDFS-5357 > URL: https://issues.apache.org/jira/browse/HDFS-5357 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.9 >Reporter: Robert Parker >Assignee: Robert Parker > > junit.framework.AssertionFailedError: Expected Exception: ServiceException > got: ExceptionInInitializerError > at junit.framework.Assert.fail(Assert.java:47) > at > org.apache.hadoop.test.TestExceptionHelper$1.evaluate(TestExceptionHelper.java:56) > at > org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) > at org.junit.runners.ParentRunner.run(ParentRunner.java:236) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208) > at > org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159) > at > org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5357) TestFileSystemAccessService failures in JDK7
Robert Parker created HDFS-5357: --- Summary: TestFileSystemAccessService failures in JDK7 Key: HDFS-5357 URL: https://issues.apache.org/jira/browse/HDFS-5357 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.9 Reporter: Robert Parker junit.framework.AssertionFailedError: Expected Exception: ServiceException got: ExceptionInInitializerError at junit.framework.Assert.fail(Assert.java:47) at org.apache.hadoop.test.TestExceptionHelper$1.evaluate(TestExceptionHelper.java:56) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5358) add 'replication' field to PathBasedCacheDirective
Colin Patrick McCabe created HDFS-5358: -- Summary: add 'replication' field to PathBasedCacheDirective Key: HDFS-5358 URL: https://issues.apache.org/jira/browse/HDFS-5358 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: HDFS-4949 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Add a 'replication' field to PathBasedCacheDirective, so that administrators can configure how many cached replicas of a block the cluster should try to maintain. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5324) Make Namespace implementation pluggable in the namenode
[ https://issues.apache.org/jira/browse/HDFS-5324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794398#comment-13794398 ] Milind Bhandarkar commented on HDFS-5324: - Colin, As I have replied to Sanjay, I am not proposing a stable API within HDFS for implementing Namespaces, and similar to what you pointed out quoting from linux kernel mailing list, the proposed API will evolve rapidly, and will change incompatibly. I have participated in at least one project to implement a HDFS protocol-compatible external file-system, and found that it was possible to do to make other proprietary (or open) file systems "appear" as if they are HDFS (not merely Hadoop-compatible). However, it does not allow the HDFS community itself to experiment with different namespace implementations, within HDFS. The primary audience for the pluggability proposal is, HDFS developers, who want to develop more scalable implementations of namespace, without having to rewrite the entire namenode code. I think Bobby Evans made the point much better than I could, on the mailing list: All the approaches for scaling namespace is based on tradeoffs. In order for evaluating the right mix of complexity, scalability, and ease of use and deployment, we need to first allow experimentation, and independent parallel development. That is what this pluggability proposal is aimed at doing. > Make Namespace implementation pluggable in the namenode > --- > > Key: HDFS-5324 > URL: https://issues.apache.org/jira/browse/HDFS-5324 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.1.1-beta > Environment: All >Reporter: Milind Bhandarkar >Assignee: Milind Bhandarkar > Fix For: 3.0.0 > > Attachments: AbstractNamesystem.java > > > For the last couple of months, we have been working on making Namespace > implementation in the namenode pluggable. We have demonstrated that it can > be done without major surgery on the namenode, and does not have noticeable > performance impact. We would like to contribute it back to Apache HDFS. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5342) Provide more information in the JMX interfaces
[
https://issues.apache.org/jira/browse/HDFS-5342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794389#comment-13794389
]
Hadoop QA commented on HDFS-5342:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12608305/HDFS-5342.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5181//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5181//console
This message is automatically generated.
> Provide more information in the JMX interfaces
> --
>
> Key: HDFS-5342
> URL: https://issues.apache.org/jira/browse/HDFS-5342
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-5342.000.patch, HDFS-5342.001.patch,
> HDFS-5342.002.patch
>
>
> Today the JMX interface in HDFS only provides a subset of information listed
> in the WebUI. This jira proposes to augment the JMX interface so that it
> provides as much information as the Web UI today.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5276) FileSystem.Statistics got performance issue on multi-thread read/write.
[
https://issues.apache.org/jira/browse/HDFS-5276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794380#comment-13794380
]
Andrew Wang commented on HDFS-5276:
---
I really like how this has shaped up. The microbenchmark numbers here very
impressive (and probably even better on more cores); thanks for doing this
Binglin. It'd still be good to see this with the actual Spark workload if
possible.
Colin, overall patch looks good, only some minor comments:
* It'd be nice to include "aggregation" somehow in the name of
{{StatisticsVIsitor}} and/or {{#visitAll}}, e.g.
{{StatisticsAggregationVisitor}} and/or {{#aggregate}}, since {{total}} is part
of the interface.
* {{FileSystem#reset}} is now resetting all the statistics, not just bytes
written and read. This makes sense to me. It looks like the current behavior is
just an oversight from when the additional statistics were added in
HADOOP-6859; maybe [~sureshms] as the original author can comment.
* Resetting by taking the current total, {{#negate}}-ing, then adding is not
immediately obvious to the reader. Is there a reason we don't just have a
{{#reset}} method and reset each {{StatisticData}} in turn?
* I'd also prefer just to not reuse the visitor / {{visitAll}} here too, since
it's kind of weird semantically: {{#total()}} mutates the {{total}}
{{StatisticsData}} member, and it depends on {{#total()}} being called exactly
once at the end.
> FileSystem.Statistics got performance issue on multi-thread read/write.
> ---
>
> Key: HDFS-5276
> URL: https://issues.apache.org/jira/browse/HDFS-5276
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.4-alpha
>Reporter: Chengxiang Li
>Assignee: Colin Patrick McCabe
> Attachments: DisableFSReadWriteBytesStat.patch, HDFS-5276.001.patch,
> HDFS-5276.002.patch, HDFSStatisticTest.java, hdfs-test.PNG, jstack-trace.PNG,
> TestFileSystemStatistics.java, ThreadLocalStat.patch
>
>
> FileSystem.Statistics is a singleton variable for each FS scheme, each
> read/write on HDFS would lead to a AutomicLong.getAndAdd(). AutomicLong does
> not perform well in multi-threads(let's say more than 30 threads). so it may
> cause serious performance issue. during our spark test profile, 32 threads
> read data from HDFS, about 70% cpu time is spent on
> FileSystem.Statistics.incrementBytesRead().
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5349) DNA_CACHE and DNA_UNCACHE should be by blockId only
[ https://issues.apache.org/jira/browse/HDFS-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5349: --- Attachment: HDFS-5349-caching.002.patch Here is an updated patch that removes genstamp, length, etc. from the DNA_CACHE and DNA_UNCACHE messages sent over the wire. Removing it from the cached block report will be a bit more work, so I'd like to wait until 5096 lands. > DNA_CACHE and DNA_UNCACHE should be by blockId only > > > Key: HDFS-5349 > URL: https://issues.apache.org/jira/browse/HDFS-5349 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: HDFS-4949 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-5349-caching.001.patch, HDFS-5349-caching.002.patch > > > DNA_CACHE and DNA_UNCACHE should be by blockId only. We don't need length > and genstamp to know what the NN asked us to cache. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5130) Add test for snapshot related FsShell and DFSAdmin commands
[ https://issues.apache.org/jira/browse/HDFS-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13794344#comment-13794344 ] Jing Zhao commented on HDFS-5130: - The patch looks good to me. Some minors: # Please add some javadoc for TestSnapshotCommands and list commands that are tested. # in toolRun(), we need to close out. # It may be better to split the testSnapshotCommands method into multiple shorter tests, and we only test one command in each test. This will also make it easier for us to add new test cases in the future. > Add test for snapshot related FsShell and DFSAdmin commands > --- > > Key: HDFS-5130 > URL: https://issues.apache.org/jira/browse/HDFS-5130 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Binglin Chang >Assignee: Binglin Chang >Priority: Minor > Attachments: HDFS-5130.v1.patch, HDFS-5130.v2.patch > > > Currently, those commands do not have tests which lead to some bugs recently. > It's better to add end-to-end tests for those commands. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5356) MiniDFSCluster shoud close all open FileSystems when shutdown()
[ https://issues.apache.org/jira/browse/HDFS-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] haosdent updated HDFS-5356: --- Status: Patch Available (was: Open) > MiniDFSCluster shoud close all open FileSystems when shutdown() > --- > > Key: HDFS-5356 > URL: https://issues.apache.org/jira/browse/HDFS-5356 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: haosdent >Priority: Critical > Attachments: HDFS-5356.patch > > > After add some metrics functions to DFSClient, I found that some unit tests > relates to metrics are failed. Because MiniDFSCluster are never close open > FileSystems, DFSClients are alive after MiniDFSCluster shutdown(). The > metrics of DFSClients in DefaultMetricsSystem are still exist and this make > other unit tests failed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5356) MiniDFSCluster shoud close all open FileSystems when shutdown()
[ https://issues.apache.org/jira/browse/HDFS-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] haosdent updated HDFS-5356: --- Description: After add some metrics functions to DFSClient, I found that some unit tests relates to metrics are failed. Because MiniDFSCluster are never close open FileSystems, DFSClients are alive after MiniDFSCluster shutdown(). The metrics of DFSClients in DefaultMetricsSystem are still exist and this make other unit tests failed. > MiniDFSCluster shoud close all open FileSystems when shutdown() > --- > > Key: HDFS-5356 > URL: https://issues.apache.org/jira/browse/HDFS-5356 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: haosdent >Priority: Critical > > After add some metrics functions to DFSClient, I found that some unit tests > relates to metrics are failed. Because MiniDFSCluster are never close open > FileSystems, DFSClients are alive after MiniDFSCluster shutdown(). The > metrics of DFSClients in DefaultMetricsSystem are still exist and this make > other unit tests failed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HDFS-5356) MiniDFSCluster shoud close all open FileSystems when shutdown()
[ https://issues.apache.org/jira/browse/HDFS-5356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] haosdent updated HDFS-5356: --- Attachment: HDFS-5356.patch > MiniDFSCluster shoud close all open FileSystems when shutdown() > --- > > Key: HDFS-5356 > URL: https://issues.apache.org/jira/browse/HDFS-5356 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: haosdent >Priority: Critical > Attachments: HDFS-5356.patch > > > After add some metrics functions to DFSClient, I found that some unit tests > relates to metrics are failed. Because MiniDFSCluster are never close open > FileSystems, DFSClients are alive after MiniDFSCluster shutdown(). The > metrics of DFSClients in DefaultMetricsSystem are still exist and this make > other unit tests failed. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5356) MiniDFSCluster shoud close all open FileSystems when shutdown()
haosdent created HDFS-5356: -- Summary: MiniDFSCluster shoud close all open FileSystems when shutdown() Key: HDFS-5356 URL: https://issues.apache.org/jira/browse/HDFS-5356 Project: Hadoop HDFS Issue Type: Bug Reporter: haosdent Priority: Critical -- This message was sent by Atlassian JIRA (v6.1#6144)
