[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749141#action_12749141
]
Hudson commented on HDFS-492:
-
Integrated in Hadoop-Hdfs-trunk #65 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/65/])
. Add two JSON JSP pages to the Namenode for providing corrupt
blocks/replicas information. Contributed by Bill Zeller
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
> Fix For: 0.21.0
>
> Attachments: hdfs-492-10.patch, hdfs-492-11.patch, hdfs-492-13.patch,
> hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch, hdfs-492-9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748932#action_12748932
]
Hudson commented on HDFS-492:
-
Integrated in Hadoop-Hdfs-trunk-Commit #5 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/5/])
. Add two JSON JSP pages to the Namenode for providing corrupt
blocks/replicas information. Contributed by Bill Zeller
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
> Fix For: 0.21.0
>
> Attachments: hdfs-492-10.patch, hdfs-492-11.patch, hdfs-492-13.patch,
> hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch, hdfs-492-9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748926#action_12748926
]
Tsz Wo (Nicholas), SZE commented on HDFS-492:
-
I forgot to say that the failed test, TestBackupNode, is not related to this.
See HDFS-192.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
> Fix For: 0.21.0
>
> Attachments: hdfs-492-10.patch, hdfs-492-11.patch, hdfs-492-13.patch,
> hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch, hdfs-492-9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748908#action_12748908
]
Suresh Srinivas commented on HDFS-492:
--
Thanks Bill for addressing all the comments. +1 for the patch.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-10.patch, hdfs-492-11.patch, hdfs-492-13.patch,
> hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch, hdfs-492-9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748738#action_12748738
]
Hadoop QA commented on HDFS-492:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12417941/hdfs-492-13.patch
against trunk revision 808670.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/97/console
This message is automatically generated.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-10.patch, hdfs-492-11.patch, hdfs-492-13.patch,
> hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch, hdfs-492-9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748605#action_12748605
]
Hadoop QA commented on HDFS-492:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12417928/hdfs-492-11.patch
against trunk revision 808193.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/94/console
This message is automatically generated.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-10.patch, hdfs-492-11.patch, hdfs-492-4.patch,
> hdfs-492-5.patch, hdfs-492-8.patch, hdfs-492-9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748473#action_12748473
]
Tsz Wo (Nicholas), SZE commented on HDFS-492:
-
In our [Coding Style|http://wiki.apache.org/hadoop/CodeReviewChecklist], "{"
should be place in the same line. For examples,
{code}
+ public long[] getCorruptReplicaBlockIds(int numExpectedBlocks,
+ Long startingBlockId)
+ {
{code}
should be
{code}
+ public long[] getCorruptReplicaBlockIds(int numExpectedBlocks,
+ Long startingBlockId) {
{code}
Also, could you remove the tab characters? We do not use tabs.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch,
> hdfs-492-9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748465#action_12748465
]
Bill Zeller commented on HDFS-492:
--
The failed test is unrelated to this patch. See:
https://issues.apache.org/jira/browse/HDFS-568
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch,
> hdfs-492-9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748264#action_12748264
]
Hadoop QA commented on HDFS-492:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12417821/hdfs-492-9.patch
against trunk revision 808193.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/92/console
This message is automatically generated.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch,
> hdfs-492-9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748189#action_12748189
]
Suresh Srinivas commented on HDFS-492:
--
+1 for the patch
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch,
> hdfs-492-9.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747753#action_12747753
]
Bill Zeller commented on HDFS-492:
--
In regards to (1), doesn't toArray() require an object array? I don't believe
this method supports primitive data types.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747732#action_12747732
]
Suresh Srinivas commented on HDFS-492:
--
# In CorruptReplicasMap.getCorruptReplicaBlockIds() you can return array by
code:
{noformat}
return corruptReplicaBlockIds.toArray(new long[corruptReplicasBlockIds.size();
{noformat}
# Change methods {{TestCorruptReplicaInfo.getBlock()}} to private
# block_info_xml.jsp - {{XMLBlockInfo.getLocalPath()}} use StringBuilder
instead of StringBuffer
# block_info_xml.jsp, corrupt_replicas_xml.jsp - could you please add comments
to indicate what these jsps do
# XMLBlockInfo constructor - should fsn be set to the parameter to constructor
irrespective of whether blockId argument is null or not?
# Rename {{XMLBlockInfo.getLocalPath()}} appropriately to indicate it is the
parent directory that the method returns
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch, hdfs-492-8.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739258#action_12739258
]
Bill Zeller commented on HDFS-492:
--
The second argument is the startingBlockId, not the starting block offset.
Passing 0 indicates a block id of 0. This function could be special cased to
treat 0 to mean to start from the beginning and assume that no real block id
would ever have the value 0, but this seems to be dirtier than just using
{{null}}.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12739243#action_12739243
]
Suresh Srinivas commented on HDFS-492:
--
long passed to the method can be set to 0 right to start from the beginning?
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12738752#action_12738752
]
Bill Zeller commented on HDFS-492:
--
I made all the above changes except for (4).
The reason I take a Long as the startingBlockId parameter (instead of a long),
is to allow {{null}} to be passed. If {{null}} is passed as startingBlockId,
the iteration begins at the beginning of the corrupt replica list. If {{null}}
is not passed, the iteration begins at the block following startingBlockId. I
couldn't think of a clean way to do this other than to create another method
which duplicates some of the functionality. Another option would be to create a
third method which takes an iterator and n and returns up to n blocks wherever
that iterator begins. Two methods could be created calling this third method
(one which advanced the iterator to the right block and the other which passed
an iterator which hadn't been advanced at all). Adding another method would
require propagating it to BlockManager and FSNameSystem, which I thought might
be too heavyweight.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737324#action_12737324
]
Suresh Srinivas commented on HDFS-492:
--
Comments on the previous version of the patch. I will review the next version
that works with trunk:
# {{BlockManager.numBlocksWithCorruptReplicas()}}
#* Currently this can be obtained by by looking
{{BlockManager.corruptReplicaBlocksCount}} (unsynchronized). However this count
is updated every {{dfs.replication.interva}}. If that approximate number is
sufficent, then there is no need to introduce this method. Also existing method
{{FSNamesystem.getCorruptReplicaBlocks()}} instead of new method added
{{FSNamesystem.numBlocksWithCorruptReplicas()}}, can be used.
#* If the number needs to be accurate, then remove
{{BlockManager.corruptReplicaBlocksCount}} and synchronize the method
{{FSNamesystem.getCorruptReplicaBlocks()}} and use
{{BlockManager.numBlocksWithCorruptReplicas()}} to get the count.
# {{CorruptReplicasMap.numBlocksWithCorruptReplicas()}} - instead of adding
this, use existing method {{CorruptReplicasMap.size()}}
# Comments in {{CorruptReplicasMap.java}} can be shorted. No need to explain
how the code works - a brief comment will be sufficient . Also remove the
comment with XXXBZ
# {{CorruptReplicasMap.getCorruptReplicaBlockIds}} - it is better to take
{{long}} as second param instead of object {{Long}}. Also returning {{long[]}}
is better than {{List}}
# TestCorruptReplicaInfo.java - use assertEquals() instead of assertTrue().
When the test fails it gives information about expected Vs actual value.
# TestCorruptReplicaInfo.block_map - can just be a HashMap
Nits:
# Generic - instead of {{n}}, use {{numExpectedBlocks}}
# CorruptReplicasMap.java - better to change the type of corruptReplicasMap
from TreeMap to SortedMap
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737192#action_12737192
]
Bill Zeller commented on HDFS-492:
--
The change to BlockInfo broke this patch. I need to make it work with the
current code.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737041#action_12737041
]
Hadoop QA commented on HDFS-492:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12414283/hdfs-492-5.patch
against trunk revision 799146.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to cause Findbugs to fail.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/32/testReport/
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/32/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/32/console
This message is automatically generated.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Assignee: Bill Zeller
>Priority: Minor
> Fix For: 0.21.0
>
> Attachments: hdfs-492-4.patch, hdfs-492-5.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732841#action_12732841
]
Hadoop QA commented on HDFS-492:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12413885/hdfs-492-4.patch
against trunk revision 794953.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 3 new Findbugs warnings.
-1 release audit. The applied patch generated 283 release audit warnings
(more than the trunk's current 282 warnings).
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/testReport/
Release audit warnings:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/24/console
This message is automatically generated.
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Priority: Minor
> Attachments: hdfs-492-4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-492) Expose corrupt replica/block information
[
https://issues.apache.org/jira/browse/HDFS-492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732801#action_12732801
]
dhruba borthakur commented on HDFS-492:
---
This utility will be be a great help to monitor and process corrupt blocks in
the file system.
Is it possible to also make HDFS throw a CorruptBlockException (which can be a
subclass of IOException) when an app tries to read a block that is corrupt.
This helps in developing intelligent clients that can do a variety of things
for this type of problem: maybe retrieve the file from an external location
(maybe from tape) or from an alternate datacenter (if it is replicated there).
> Expose corrupt replica/block information
>
>
> Key: HDFS-492
> URL: https://issues.apache.org/jira/browse/HDFS-492
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, name-node
>Affects Versions: 0.21.0
>Reporter: Bill Zeller
>Priority: Minor
> Attachments: hdfs-492-4.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This adds two additional functions to FSNamesystem to provide more
> information about corrupt replicas. It also adds two servlets to the namenode
> that provide information (in JSON) about all blocks with corrupt replicas as
> well as information about a specific block. It also changes the file browsing
> servlet by adding a link from block ids to the above mentioned block
> information page.
> These JSON pages are designed to be used by client side tools which wish to
> analyze corrupt block/replicas. The only change to an existing (non-servlet)
> class is described below.
> Currently, CorruptReplicasMap stores a map of corrupt replica information and
> allows insertion and deletion. It also gives information about the corrupt
> replicas for a specific block. It does not allow iteration over all corrupt
> blocks. Two additional functions will be added to FSNamesystem (which will
> call BlockManager which will call CorruptReplicasMap). The first will return
> the size of the corrupt replicas map, which represents the number of blocks
> that have corrupt replicas (but less than the number of corrupt replicas if a
> block has multiple corrupt replicas). The second will allow "paging" through
> a list of block ids that contain corrupt replicas:
> {{public synchronized List getCorruptReplicaBlockIds(int n, Long
> startingBlockId)}}
> {{n}} is the number of block ids to return and {{startingBlockId}} is the
> block id offset. To prevent a large number of items being returned at one
> time, n is constrained to 0 <= {{n}} <= 100. If {{startingBlockId}} is null,
> up to {{n}} items are returned starting at the beginning of the list.
> Ordering is enforced through the internal use of TreeMap in
> CorruptReplicasMap.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
