Re: Application hangs in initializing HDFS client inside an AXIS webservice under JBoss 7 environment
ok Best Regards, === Chun Tee Yuen, 曾迪元, PMP PI Sect, F6MITD Taiwan Semiconductor Manufacturing Company, Ltd. Tel: 886-6-5059688 Ext: 7148239 === --- TSMC PROPERTY This email communication (and any attachments) is proprietary information for the sole use of its intended recipient. Any unauthorized review, use or distribution by anyone other than the intended recipient is strictly prohibited. If you are not the intended recipient, please notify the sender by replying to this email, and then delete this email and any copies of it immediately. Thank you. ---
Application hangs in initializing HDFS client inside an AXIS webservice under JBoss 7 environment
Hi All, I have been trying to initialize an HDFS client inside an AXIS webservice under JBoss 7 environment, but the application hangs without any exception throwing out. The problem seems to be happenned in Filesystem.java from Hadoop-0.20.2-cdh3u0 at line 1668: this.ugi = UserGroupInformation.getCurrentUser(); Any idea? Thanks in advance! Best Regards, === Chun Tee Yuen, 曾迪元, PMP PI Sect, F6MITD Taiwan Semiconductor Manufacturing Company, Ltd. Tel: 886-6-5059688 Ext: 7148239 === --- TSMC PROPERTY This email communication (and any attachments) is proprietary information for the sole use of its intended recipient. Any unauthorized review, use or distribution by anyone other than the intended recipient is strictly prohibited. If you are not the intended recipient, please notify the sender by replying to this email, and then delete this email and any copies of it immediately. Thank you. ---
[jira] [Commented] (HDFS-2308) NamenodeProtocol.endCheckpoint is vestigial and can be removed
[ https://issues.apache.org/jira/browse/HDFS-2308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138036#comment-13138036 ] Hadoop QA commented on HDFS-2308: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501219/hdfs-2308-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestFileCreationClient org.apache.hadoop.hdfs.TestSetrepIncreasing +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1482//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1482//console This message is automatically generated. > NamenodeProtocol.endCheckpoint is vestigial and can be removed > -- > > Key: HDFS-2308 > URL: https://issues.apache.org/jira/browse/HDFS-2308 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Eli Collins > Attachments: hdfs-2308-1.patch, hdfs-2308-1.patch > > > Post HDFS-1073, this method is now a no-op. All it does currently is validate > the CheckpointSignature provided, but does nothing to mutate NN state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1108) Log newly allocated blocks
[ https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137976#comment-13137976 ] Hadoop QA commented on HDFS-1108: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501211/hdfs-1108-habranch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1483//console This message is automatically generated. > Log newly allocated blocks > -- > > Key: HDFS-1108 > URL: https://issues.apache.org/jira/browse/HDFS-1108 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: dhruba borthakur >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, > hdfs-1108-habranch.txt, hdfs-1108.txt > > > The current HDFS design says that newly allocated blocks for a file are not > persisted in the NN transaction log when the block is allocated. Instead, a > hflush() or a close() on the file persists the blocks into the transaction > log. It would be nice if we can immediately persist newly allocated blocks > (as soon as they are allocated) for specific files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2371) Refactor BlockSender.java for better readability
[ https://issues.apache.org/jira/browse/HDFS-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137967#comment-13137967 ] Hudson commented on HDFS-2371: -- Integrated in Hadoop-Common-trunk-Commit #1182 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1182/]) Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in CHANGES.txt. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Refactor BlockSender.java for better readability > > > Key: HDFS-2371 > URL: https://issues.apache.org/jira/browse/HDFS-2371 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2371.txt, HDFS-2371.txt, HDFS-2371.txt > > > BlockSender.java is hard to read and understand. I propose refactoring it for > better readability -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2479) HDFS Client Data Types in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137963#comment-13137963 ] Hudson commented on HDFS-2479: -- Integrated in Hadoop-Common-trunk-Commit #1182 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1182/]) HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay) sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto > HDFS Client Data Types in Protocol Buffers > -- > > Key: HDFS-2479 > URL: https://issues.apache.org/jira/browse/HDFS-2479 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: hdfs.proto, hdfs.proto, hdfs.proto, > pbClientTypes1.patch, pbClientTypes2.patch, pbClientTypes3.patch, > pbClientTypes4.patch, pbClientTypes5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137957#comment-13137957
]
Hudson commented on HDFS-2298:
--
Integrated in Hadoop-Common-trunk-Commit #1182 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1182/])
Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in
CHANGES.txt.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2355) Federation: enable using the same configuration file across all the nodes in the cluster.
[ https://issues.apache.org/jira/browse/HDFS-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137955#comment-13137955 ] Hudson commented on HDFS-2355: -- Integrated in Hadoop-Common-trunk-Commit #1182 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1182/]) Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in CHANGES.txt. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Federation: enable using the same configuration file across all the nodes in > the cluster. > - > > Key: HDFS-2355 > URL: https://issues.apache.org/jira/browse/HDFS-2355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2355.txt, HDFS-2355.txt > > > In federation the configuration files xml include another configuration xml. > This configuration xml is empty for datanodes and defines nameservice ID for > namenodes, secondary and backup nodes. The need for xml include can be > eliminated by namenodes determining the nameservice ID by matching its local > address with a configured address (such as namenode rpc address) and on match > using the nameservice ID of the corresponding configuration parameter. > For more details about federation configuration see > https://issues.apache.org/jira/browse/HDFS-1689?focusedCommentId=13001749&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13001749 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2479) HDFS Client Data Types in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137942#comment-13137942 ] Hudson commented on HDFS-2479: -- Integrated in Hadoop-Hdfs-trunk-Commit #1259 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1259/]) HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay) sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto > HDFS Client Data Types in Protocol Buffers > -- > > Key: HDFS-2479 > URL: https://issues.apache.org/jira/browse/HDFS-2479 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: hdfs.proto, hdfs.proto, hdfs.proto, > pbClientTypes1.patch, pbClientTypes2.patch, pbClientTypes3.patch, > pbClientTypes4.patch, pbClientTypes5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2371) Refactor BlockSender.java for better readability
[ https://issues.apache.org/jira/browse/HDFS-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137946#comment-13137946 ] Hudson commented on HDFS-2371: -- Integrated in Hadoop-Hdfs-trunk-Commit #1259 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1259/]) Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in CHANGES.txt. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Refactor BlockSender.java for better readability > > > Key: HDFS-2371 > URL: https://issues.apache.org/jira/browse/HDFS-2371 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2371.txt, HDFS-2371.txt, HDFS-2371.txt > > > BlockSender.java is hard to read and understand. I propose refactoring it for > better readability -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2355) Federation: enable using the same configuration file across all the nodes in the cluster.
[ https://issues.apache.org/jira/browse/HDFS-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137934#comment-13137934 ] Hudson commented on HDFS-2355: -- Integrated in Hadoop-Hdfs-trunk-Commit #1259 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1259/]) Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in CHANGES.txt. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Federation: enable using the same configuration file across all the nodes in > the cluster. > - > > Key: HDFS-2355 > URL: https://issues.apache.org/jira/browse/HDFS-2355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2355.txt, HDFS-2355.txt > > > In federation the configuration files xml include another configuration xml. > This configuration xml is empty for datanodes and defines nameservice ID for > namenodes, secondary and backup nodes. The need for xml include can be > eliminated by namenodes determining the nameservice ID by matching its local > address with a configured address (such as namenode rpc address) and on match > using the nameservice ID of the corresponding configuration parameter. > For more details about federation configuration see > https://issues.apache.org/jira/browse/HDFS-1689?focusedCommentId=13001749&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13001749 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137936#comment-13137936
]
Hudson commented on HDFS-2298:
--
Integrated in Hadoop-Hdfs-trunk-Commit #1259 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1259/])
Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in
CHANGES.txt.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise readahead and drop-behind
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137926#comment-13137926 ] Hadoop QA commented on HDFS-2465: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501194/hdfs-2465.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks org.apache.hadoop.hdfs.TestSmallBlock org.apache.hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling org.apache.hadoop.hdfs.server.balancer.TestBalancer org.apache.hadoop.hdfs.TestFileCreation org.apache.hadoop.hdfs.TestFileCreationClient org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks org.apache.hadoop.hdfs.TestDFSClientExcludedNodes org.apache.hadoop.hdfs.TestSetrepIncreasing org.apache.hadoop.hdfs.TestPread +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1480//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1480//console This message is automatically generated. > Add HDFS support for fadvise readahead and drop-behind > -- > > Key: HDFS-2465 > URL: https://issues.apache.org/jira/browse/HDFS-2465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt > > > This is the HDFS side of HADOOP-7714. The initial implementation is heuristic > based and should be considered experimental, as discussed in the parent JIRA. > It should be off by default until better heuristics, APIs, and tuning > experience is developed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2308) NamenodeProtocol.endCheckpoint is vestigial and can be removed
[ https://issues.apache.org/jira/browse/HDFS-2308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2308: -- Attachment: hdfs-2308-1.patch Right patch this time. > NamenodeProtocol.endCheckpoint is vestigial and can be removed > -- > > Key: HDFS-2308 > URL: https://issues.apache.org/jira/browse/HDFS-2308 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Eli Collins > Attachments: hdfs-2308-1.patch, hdfs-2308-1.patch > > > Post HDFS-1073, this method is now a no-op. All it does currently is validate > the CheckpointSignature provided, but does nothing to mutate NN state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2308) NamenodeProtocol.endCheckpoint is vestigial and can be removed
[ https://issues.apache.org/jira/browse/HDFS-2308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2308: -- Target Version/s: 0.24.0 Status: Patch Available (was: Open) > NamenodeProtocol.endCheckpoint is vestigial and can be removed > -- > > Key: HDFS-2308 > URL: https://issues.apache.org/jira/browse/HDFS-2308 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Eli Collins > Attachments: hdfs-2308-1.patch, hdfs-2308-1.patch > > > Post HDFS-1073, this method is now a no-op. All it does currently is validate > the CheckpointSignature provided, but does nothing to mutate NN state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2371) Refactor BlockSender.java for better readability
[ https://issues.apache.org/jira/browse/HDFS-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137886#comment-13137886 ] Hudson commented on HDFS-2371: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1200 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1200/]) Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in CHANGES.txt. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Refactor BlockSender.java for better readability > > > Key: HDFS-2371 > URL: https://issues.apache.org/jira/browse/HDFS-2371 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2371.txt, HDFS-2371.txt, HDFS-2371.txt > > > BlockSender.java is hard to read and understand. I propose refactoring it for > better readability -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137885#comment-13137885
]
Hudson commented on HDFS-2298:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #1200 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1200/])
Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in
CHANGES.txt.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2479) HDFS Client Data Types in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137883#comment-13137883 ] Hudson commented on HDFS-2479: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1200 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1200/]) HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay) sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto > HDFS Client Data Types in Protocol Buffers > -- > > Key: HDFS-2479 > URL: https://issues.apache.org/jira/browse/HDFS-2479 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: hdfs.proto, hdfs.proto, hdfs.proto, > pbClientTypes1.patch, pbClientTypes2.patch, pbClientTypes3.patch, > pbClientTypes4.patch, pbClientTypes5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2355) Federation: enable using the same configuration file across all the nodes in the cluster.
[ https://issues.apache.org/jira/browse/HDFS-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137884#comment-13137884 ] Hudson commented on HDFS-2355: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1200 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1200/]) Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in CHANGES.txt. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Federation: enable using the same configuration file across all the nodes in > the cluster. > - > > Key: HDFS-2355 > URL: https://issues.apache.org/jira/browse/HDFS-2355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2355.txt, HDFS-2355.txt > > > In federation the configuration files xml include another configuration xml. > This configuration xml is empty for datanodes and defines nameservice ID for > namenodes, secondary and backup nodes. The need for xml include can be > eliminated by namenodes determining the nameservice ID by matching its local > address with a configured address (such as namenode rpc address) and on match > using the nameservice ID of the corresponding configuration parameter. > For more details about federation configuration see > https://issues.apache.org/jira/browse/HDFS-1689?focusedCommentId=13001749&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13001749 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2371) Refactor BlockSender.java for better readability
[ https://issues.apache.org/jira/browse/HDFS-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137875#comment-13137875 ] Hudson commented on HDFS-2371: -- Integrated in Hadoop-Mapreduce-0.23-Build #65 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/65/]) svn merge -c 1177161 from trunk for HDFS-2371. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189981 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java > Refactor BlockSender.java for better readability > > > Key: HDFS-2371 > URL: https://issues.apache.org/jira/browse/HDFS-2371 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2371.txt, HDFS-2371.txt, HDFS-2371.txt > > > BlockSender.java is hard to read and understand. I propose refactoring it for > better readability -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2432) webhdfs setreplication api should return a 403 when called on a directory
[
https://issues.apache.org/jira/browse/HDFS-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137863#comment-13137863
]
Hudson commented on HDFS-2432:
--
Integrated in Hadoop-Mapreduce-0.23-Build #65 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/65/])
svn merge -c 1190077 from trunk for HDFS-2432.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190082
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
> webhdfs setreplication api should return a 403 when called on a directory
> -
>
> Key: HDFS-2432
> URL: https://issues.apache.org/jira/browse/HDFS-2432
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: h2432_20111026.patch, h2432_20111026_0.20s.patch,
> h2432_20111027.patch, h2432_20111027_0.20s.patch
>
>
> Currently the set replication api on a directory leads to a 200.
> Request URI
> http://NN:50070/webhdfs/tmp/webhdfs_data/dir_replication_tests?op=SETREPLICATION&replication=5
> Request Method: PUT
> Status Line: HTTP/1.1 200 OK
> Response Content: {"boolean":false}
> Since we can determine that this call did not succeed (boolean=false) we
> should rather just return a 403
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137862#comment-13137862
]
Hudson commented on HDFS-2298:
--
Integrated in Hadoop-Mapreduce-0.23-Build #65 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/65/])
svn merge -c 1177757 from trunk for HDFS-2298.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189983
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2355) Federation: enable using the same configuration file across all the nodes in the cluster.
[ https://issues.apache.org/jira/browse/HDFS-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137860#comment-13137860 ] Hudson commented on HDFS-2355: -- Integrated in Hadoop-Mapreduce-0.23-Build #65 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/65/]) svn merge -c 1177100 from trunk for HDFS-2355. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189979 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestMulitipleNNDataBlockScanner.java > Federation: enable using the same configuration file across all the nodes in > the cluster. > - > > Key: HDFS-2355 > URL: https://issues.apache.org/jira/browse/HDFS-2355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2355.txt, HDFS-2355.txt > > > In federation the configuration files xml include another configuration xml. > This configuration xml is empty for datanodes and defines nameservice ID for > namenodes, secondary and backup nodes. The need for xml include can be > eliminated by namenodes determining the nameservice ID by matching its local > address with a configured address (such as namenode rpc address) and on match > using the nameservice ID of the corresponding configuration parameter. > For more details about federation configuration see > https://issues.apache.org/jira/browse/HDFS-1689?focusedCommentId=13001749&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13001749 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137861#comment-13137861 ] Hudson commented on HDFS-2500: -- Integrated in Hadoop-Mapreduce-0.23-Build #65 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/65/]) HDFS-2500. Avoid file system operations in BPOfferService thread while processing deletes. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190072 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1108) Log newly allocated blocks
[ https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-1108: -- Attachment: hdfs-1108-habranch.txt Updated patch attached for the HDFS-1623 branch. It removes the new configuration option - only append and shared-edits-dir based HA, using the config flag introduced in HDFS-1975 - trigger block persistence. In shared-edits-dir based HA the log and sync are needed to ensure the standby state is consistent in the case of fail-over. This does not preclude adding persistent, guaranteed streams in the future, and introduces no additional overhead for the BN-based approach. Similarly, we can unconditionally log blocks in the future if benchmarking doesn't show significant impact. Reasonable? > Log newly allocated blocks > -- > > Key: HDFS-1108 > URL: https://issues.apache.org/jira/browse/HDFS-1108 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: dhruba borthakur >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, > hdfs-1108-habranch.txt, hdfs-1108.txt > > > The current HDFS design says that newly allocated blocks for a file are not > persisted in the NN transaction log when the block is allocated. Instead, a > hflush() or a close() on the file persists the blocks into the transaction > log. It would be nice if we can immediately persist newly allocated blocks > (as soon as they are allocated) for specific files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2479) HDFS Client Data Types in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137848#comment-13137848 ] Hudson commented on HDFS-2479: -- Integrated in Hadoop-Mapreduce-trunk #875 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/875/]) HDFS-2479 HDFS Client Data Types in Protocol Buffers (sanjay) sradia : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190127 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsProtoUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/proto/HdfsProtos.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/proto/hdfs.proto > HDFS Client Data Types in Protocol Buffers > -- > > Key: HDFS-2479 > URL: https://issues.apache.org/jira/browse/HDFS-2479 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: hdfs.proto, hdfs.proto, hdfs.proto, > pbClientTypes1.patch, pbClientTypes2.patch, pbClientTypes3.patch, > pbClientTypes4.patch, pbClientTypes5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2479) HDFS Client Data Types in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-2479: --- Resolution: Fixed Status: Resolved (was: Patch Available) > HDFS Client Data Types in Protocol Buffers > -- > > Key: HDFS-2479 > URL: https://issues.apache.org/jira/browse/HDFS-2479 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: hdfs.proto, hdfs.proto, hdfs.proto, > pbClientTypes1.patch, pbClientTypes2.patch, pbClientTypes3.patch, > pbClientTypes4.patch, pbClientTypes5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2496) Separate datatypes for DatanodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137847#comment-13137847 ] Hudson commented on HDFS-2496: -- Integrated in Hadoop-Mapreduce-trunk #874 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/874/]) HDFS-2496. Separate datatypes for DatanodeProtocol. Contributed by Suresh Srinivas. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189901 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolR23Compatible/BlockWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/BlockRecoveryCommand.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocol/DatanodeRegistration.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/BalancerBandwidthCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/BlockCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/BlockRecoveryCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/DatanodeCommandHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/DatanodeCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/DatanodeProtocolServerSideTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/DatanodeProtocolTranslatorR23.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/DatanodeRegistrationWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/DatanodeWireProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/FinalizeCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/KeyUpdateCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/ReceivedDeletedBlockInfoWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/RegisterCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/ServerCommandWritable.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/protocolR23Compatible/UpgradeCommandWritable.java > Separate datatypes for DatanodeProtocol > --- > > Key: HDFS-2496 > URL: https://issues.apache.org/jira/browse/HDFS-2496 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0, 0.24.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: HDFS-2496.txt, HDFS-2496.txt, HDFS-2496.txt > > > This jira separates for DatanodeProtocol the wire types from the types used > by the client and server, similar to HDFS-2181. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2371) Refactor BlockSender.java for better readability
[ https://issues.apache.org/jira/browse/HDFS-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137843#comment-13137843 ] Hudson commented on HDFS-2371: -- Integrated in Hadoop-Mapreduce-trunk #874 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/874/]) Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in CHANGES.txt. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Refactor BlockSender.java for better readability > > > Key: HDFS-2371 > URL: https://issues.apache.org/jira/browse/HDFS-2371 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2371.txt, HDFS-2371.txt, HDFS-2371.txt > > > BlockSender.java is hard to read and understand. I propose refactoring it for > better readability -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2507) HA: Allow saveNamespace operations to be canceled
[ https://issues.apache.org/jira/browse/HDFS-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137834#comment-13137834 ] Hudson commented on HDFS-2507: -- Integrated in Hadoop-Mapreduce-trunk #874 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/874/]) HDFS-2507. Allow saveNamespace operations to be canceled. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190060 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceCancelledException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java > HA: Allow saveNamespace operations to be canceled > - > > Key: HDFS-2507 > URL: https://issues.apache.org/jira/browse/HDFS-2507 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: HA branch (HDFS-1623), 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: hdfs-2507.txt > > > For HDFS-2291 (checkpointing in HA setup) we need to allow the StandbyNode to > handle a failover while it may be in the middle of saving a checkpoint. This > implies that checkpoints need to be abortable. > This JIRA is to implement this functionality. Even though this is only needed > for HA, I think we should do the infrastructure in trunk to simplify future > merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137829#comment-13137829 ] Hudson commented on HDFS-2500: -- Integrated in Hadoop-Mapreduce-trunk #874 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/874/]) HDFS-2500. Avoid file system operations in BPOfferService thread while processing deletes. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190071 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137830#comment-13137830
]
Hudson commented on HDFS-2298:
--
Integrated in Hadoop-Mapreduce-trunk #874 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/874/])
Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in
CHANGES.txt.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2432) webhdfs setreplication api should return a 403 when called on a directory
[
https://issues.apache.org/jira/browse/HDFS-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137831#comment-13137831
]
Hudson commented on HDFS-2432:
--
Integrated in Hadoop-Mapreduce-trunk #874 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/874/])
HDFS-2432. Webhdfs: response FORBIDDEN when setReplication on non-files;
clear umask before creating a flie; throw IllegalArgumentException if setOwner
with both owner and group empty; throw FileNotFoundException if getFileStatus
on non-existing files; fix bugs in getBlockLocations; and changed
getFileChecksum json response root to "FileChecksum".
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190077
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
> webhdfs setreplication api should return a 403 when called on a directory
> -
>
> Key: HDFS-2432
> URL: https://issues.apache.org/jira/browse/HDFS-2432
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: h2432_20111026.patch, h2432_20111026_0.20s.patch,
> h2432_20111027.patch, h2432_20111027_0.20s.patch
>
>
> Currently the set replication api on a directory leads to a 200.
> Request URI
> http://NN:50070/webhdfs/tmp/webhdfs_data/dir_replication_tests?op=SETREPLICATION&replication=5
> Request Method: PUT
> Status Line: HTTP/1.1 200 OK
> Response Content: {"boolean":false}
> Since we can determine that this call did not succeed (boolean=false) we
> should rather just return a 403
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2355) Federation: enable using the same configuration file across all the nodes in the cluster.
[ https://issues.apache.org/jira/browse/HDFS-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137828#comment-13137828 ] Hudson commented on HDFS-2355: -- Integrated in Hadoop-Mapreduce-trunk #874 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/874/]) Move HDFS-2355, HDFS-2371 and HDFS-2298 entries from trunk to 0.23 in CHANGES.txt. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190126 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Federation: enable using the same configuration file across all the nodes in > the cluster. > - > > Key: HDFS-2355 > URL: https://issues.apache.org/jira/browse/HDFS-2355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2355.txt, HDFS-2355.txt > > > In federation the configuration files xml include another configuration xml. > This configuration xml is empty for datanodes and defines nameservice ID for > namenodes, secondary and backup nodes. The need for xml include can be > eliminated by namenodes determining the nameservice ID by matching its local > address with a configured address (such as namenode rpc address) and on match > using the nameservice ID of the corresponding configuration parameter. > For more details about federation configuration see > https://issues.apache.org/jira/browse/HDFS-1689?focusedCommentId=13001749&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13001749 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2308) NamenodeProtocol.endCheckpoint is vestigial and can be removed
[ https://issues.apache.org/jira/browse/HDFS-2308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HDFS-2308: - Assignee: Eli Collins (was: Aaron T. Myers) > NamenodeProtocol.endCheckpoint is vestigial and can be removed > -- > > Key: HDFS-2308 > URL: https://issues.apache.org/jira/browse/HDFS-2308 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Eli Collins > Attachments: hdfs-2308-1.patch > > > Post HDFS-1073, this method is now a no-op. All it does currently is validate > the CheckpointSignature provided, but does nothing to mutate NN state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2308) NamenodeProtocol.endCheckpoint is vestigial and can be removed
[ https://issues.apache.org/jira/browse/HDFS-2308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2308: -- Attachment: hdfs-2308-1.patch Patch attached. Not a total nop (validates the storage info) so I just updated the method and javadoc. > NamenodeProtocol.endCheckpoint is vestigial and can be removed > -- > > Key: HDFS-2308 > URL: https://issues.apache.org/jira/browse/HDFS-2308 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: hdfs-2308-1.patch > > > Post HDFS-1073, this method is now a no-op. All it does currently is validate > the CheckpointSignature provided, but does nothing to mutate NN state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead and drop-behind
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2465: -- Attachment: hdfs-2465.txt reuploading patch to trigger build. Also fixed the style nit Eli mentioned above. > Add HDFS support for fadvise readahead and drop-behind > -- > > Key: HDFS-2465 > URL: https://issues.apache.org/jira/browse/HDFS-2465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2465.txt, hdfs-2465.txt, hdfs-2465.txt > > > This is the HDFS side of HADOOP-7714. The initial implementation is heuristic > based and should be considered experimental, as discussed in the parent JIRA. > It should be off by default until better heuristics, APIs, and tuning > experience is developed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2430) The number of failed or low-resource volumes the NN can tolerate should be configurable
[ https://issues.apache.org/jira/browse/HDFS-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137754#comment-13137754 ] Aaron T. Myers commented on HDFS-2430: -- Hold off a minute, Eli. I just realized that I completely forgot to make the number of failures in the redundant pool configurable. Let me fix that before you spend time reviewing it. > The number of failed or low-resource volumes the NN can tolerate should be > configurable > --- > > Key: HDFS-2430 > URL: https://issues.apache.org/jira/browse/HDFS-2430 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-2430.patch > > > Currently the number of failed or low-resource volumes the NN can tolerate is > effectively hard-coded at 1. It would be nice if this were configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2432) webhdfs setreplication api should return a 403 when called on a directory
[
https://issues.apache.org/jira/browse/HDFS-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137743#comment-13137743
]
Hudson commented on HDFS-2432:
--
Integrated in Hadoop-Mapreduce-0.23-Commit #84 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/84/])
svn merge -c 1190077 from trunk for HDFS-2432.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190082
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
> webhdfs setreplication api should return a 403 when called on a directory
> -
>
> Key: HDFS-2432
> URL: https://issues.apache.org/jira/browse/HDFS-2432
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: h2432_20111026.patch, h2432_20111026_0.20s.patch,
> h2432_20111027.patch, h2432_20111027_0.20s.patch
>
>
> Currently the set replication api on a directory leads to a 200.
> Request URI
> http://NN:50070/webhdfs/tmp/webhdfs_data/dir_replication_tests?op=SETREPLICATION&replication=5
> Request Method: PUT
> Status Line: HTTP/1.1 200 OK
> Response Content: {"boolean":false}
> Since we can determine that this call did not succeed (boolean=false) we
> should rather just return a 403
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2432) webhdfs setreplication api should return a 403 when called on a directory
[
https://issues.apache.org/jira/browse/HDFS-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137741#comment-13137741
]
Hudson commented on HDFS-2432:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #1196 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1196/])
HDFS-2432. Webhdfs: response FORBIDDEN when setReplication on non-files;
clear umask before creating a flie; throw IllegalArgumentException if setOwner
with both owner and group empty; throw FileNotFoundException if getFileStatus
on non-existing files; fix bugs in getBlockLocations; and changed
getFileChecksum json response root to "FileChecksum".
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190077
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
> webhdfs setreplication api should return a 403 when called on a directory
> -
>
> Key: HDFS-2432
> URL: https://issues.apache.org/jira/browse/HDFS-2432
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: h2432_20111026.patch, h2432_20111026_0.20s.patch,
> h2432_20111027.patch, h2432_20111027_0.20s.patch
>
>
> Currently the set replication api on a directory leads to a 200.
> Request URI
> http://NN:50070/webhdfs/tmp/webhdfs_data/dir_replication_tests?op=SETREPLICATION&replication=5
> Request Method: PUT
> Status Line: HTTP/1.1 200 OK
> Response Content: {"boolean":false}
> Since we can determine that this call did not succeed (boolean=false) we
> should rather just return a 403
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137737#comment-13137737 ] Hudson commented on HDFS-2500: -- Integrated in Hadoop-Mapreduce-0.23-Commit #83 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/83/]) HDFS-2500. Avoid file system operations in BPOfferService thread while processing deletes. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190072 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2430) The number of failed or low-resource volumes the NN can tolerate should be configurable
[ https://issues.apache.org/jira/browse/HDFS-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137734#comment-13137734 ] Eli Collins commented on HDFS-2430: --- Agree that option #1 is better. The notion of a NN "required" dir here parallels the notion of a "critical" dir in the DN (proposed in HDFS-1848). I'll check out the patch. > The number of failed or low-resource volumes the NN can tolerate should be > configurable > --- > > Key: HDFS-2430 > URL: https://issues.apache.org/jira/browse/HDFS-2430 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-2430.patch > > > Currently the number of failed or low-resource volumes the NN can tolerate is > effectively hard-coded at 1. It would be nice if this were configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2393) Mark appropriate methods of ClientProtocol with the idempotent annotation
[
https://issues.apache.org/jira/browse/HDFS-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-2393:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed this.
> Mark appropriate methods of ClientProtocol with the idempotent annotation
> -
>
> Key: HDFS-2393
> URL: https://issues.apache.org/jira/browse/HDFS-2393
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: HA branch (HDFS-1623)
>
> Attachments: HDFS-2393-HDFS-1623.patch, HDFS-2393-HDFS-1623.patch
>
>
> HDFS-1973 will make the {{DFSClient}} take advantage of the annotation. This
> JIRA is to identify which methods can be marked as idempotent.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2432) webhdfs setreplication api should return a 403 when called on a directory
[
https://issues.apache.org/jira/browse/HDFS-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137729#comment-13137729
]
Hudson commented on HDFS-2432:
--
Integrated in Hadoop-Common-0.23-Commit #88 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/88/])
svn merge -c 1190077 from trunk for HDFS-2432.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190082
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
> webhdfs setreplication api should return a 403 when called on a directory
> -
>
> Key: HDFS-2432
> URL: https://issues.apache.org/jira/browse/HDFS-2432
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: h2432_20111026.patch, h2432_20111026_0.20s.patch,
> h2432_20111027.patch, h2432_20111027_0.20s.patch
>
>
> Currently the set replication api on a directory leads to a 200.
> Request URI
> http://NN:50070/webhdfs/tmp/webhdfs_data/dir_replication_tests?op=SETREPLICATION&replication=5
> Request Method: PUT
> Status Line: HTTP/1.1 200 OK
> Response Content: {"boolean":false}
> Since we can determine that this call did not succeed (boolean=false) we
> should rather just return a 403
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2432) webhdfs setreplication api should return a 403 when called on a directory
[
https://issues.apache.org/jira/browse/HDFS-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137732#comment-13137732
]
Hudson commented on HDFS-2432:
--
Integrated in Hadoop-Hdfs-0.23-Commit #88 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/88/])
svn merge -c 1190077 from trunk for HDFS-2432.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190082
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
> webhdfs setreplication api should return a 403 when called on a directory
> -
>
> Key: HDFS-2432
> URL: https://issues.apache.org/jira/browse/HDFS-2432
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: h2432_20111026.patch, h2432_20111026_0.20s.patch,
> h2432_20111027.patch, h2432_20111027_0.20s.patch
>
>
> Currently the set replication api on a directory leads to a 200.
> Request URI
> http://NN:50070/webhdfs/tmp/webhdfs_data/dir_replication_tests?op=SETREPLICATION&replication=5
> Request Method: PUT
> Status Line: HTTP/1.1 200 OK
> Response Content: {"boolean":false}
> Since we can determine that this call did not succeed (boolean=false) we
> should rather just return a 403
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2432) webhdfs setreplication api should return a 403 when called on a directory
[
https://issues.apache.org/jira/browse/HDFS-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137726#comment-13137726
]
Hudson commented on HDFS-2432:
--
Integrated in Hadoop-Common-trunk-Commit #1180 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1180/])
HDFS-2432. Webhdfs: response FORBIDDEN when setReplication on non-files;
clear umask before creating a flie; throw IllegalArgumentException if setOwner
with both owner and group empty; throw FileNotFoundException if getFileStatus
on non-existing files; fix bugs in getBlockLocations; and changed
getFileChecksum json response root to "FileChecksum".
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190077
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
> webhdfs setreplication api should return a 403 when called on a directory
> -
>
> Key: HDFS-2432
> URL: https://issues.apache.org/jira/browse/HDFS-2432
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: h2432_20111026.patch, h2432_20111026_0.20s.patch,
> h2432_20111027.patch, h2432_20111027_0.20s.patch
>
>
> Currently the set replication api on a directory leads to a 200.
> Request URI
> http://NN:50070/webhdfs/tmp/webhdfs_data/dir_replication_tests?op=SETREPLICATION&replication=5
> Request Method: PUT
> Status Line: HTTP/1.1 200 OK
> Response Content: {"boolean":false}
> Since we can determine that this call did not succeed (boolean=false) we
> should rather just return a 403
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2432) webhdfs setreplication api should return a 403 when called on a directory
[
https://issues.apache.org/jira/browse/HDFS-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137723#comment-13137723
]
Hudson commented on HDFS-2432:
--
Integrated in Hadoop-Hdfs-trunk-Commit #1257 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1257/])
HDFS-2432. Webhdfs: response FORBIDDEN when setReplication on non-files;
clear umask before creating a flie; throw IllegalArgumentException if setOwner
with both owner and group empty; throw FileNotFoundException if getFileStatus
on non-existing files; fix bugs in getBlockLocations; and changed
getFileChecksum json response root to "FileChecksum".
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190077
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/resources/DatanodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsFileSystemContract.java
> webhdfs setreplication api should return a 403 when called on a directory
> -
>
> Key: HDFS-2432
> URL: https://issues.apache.org/jira/browse/HDFS-2432
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: h2432_20111026.patch, h2432_20111026_0.20s.patch,
> h2432_20111027.patch, h2432_20111027_0.20s.patch
>
>
> Currently the set replication api on a directory leads to a 200.
> Request URI
> http://NN:50070/webhdfs/tmp/webhdfs_data/dir_replication_tests?op=SETREPLICATION&replication=5
> Request Method: PUT
> Status Line: HTTP/1.1 200 OK
> Response Content: {"boolean":false}
> Since we can determine that this call did not succeed (boolean=false) we
> should rather just return a 403
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2432) webhdfs setreplication api should return a 403 when called on a directory
[
https://issues.apache.org/jira/browse/HDFS-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2432:
-
Resolution: Fixed
Fix Version/s: 0.24.0
0.23.0
0.20.206.0
0.20.205.1
Target Version/s: 0.20.205.1, 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0,
0.20.205.1)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Jitendra for the review.
The failed tests are not related.
I have committed this.
> webhdfs setreplication api should return a 403 when called on a directory
> -
>
> Key: HDFS-2432
> URL: https://issues.apache.org/jira/browse/HDFS-2432
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0
>
> Attachments: h2432_20111026.patch, h2432_20111026_0.20s.patch,
> h2432_20111027.patch, h2432_20111027_0.20s.patch
>
>
> Currently the set replication api on a directory leads to a 200.
> Request URI
> http://NN:50070/webhdfs/tmp/webhdfs_data/dir_replication_tests?op=SETREPLICATION&replication=5
> Request Method: PUT
> Status Line: HTTP/1.1 200 OK
> Response Content: {"boolean":false}
> Since we can determine that this call did not succeed (boolean=false) we
> should rather just return a 403
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2393) Mark appropriate methods of ClientProtocol with the idempotent annotation
[
https://issues.apache.org/jira/browse/HDFS-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-2393:
-
Attachment: HDFS-2393-HDFS-1623.patch
Thanks a lot for the review, Todd. Attaching patch rebased on HDFS-1623 branch.
I'm going to commit this momentarily.
> Mark appropriate methods of ClientProtocol with the idempotent annotation
> -
>
> Key: HDFS-2393
> URL: https://issues.apache.org/jira/browse/HDFS-2393
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: HA branch (HDFS-1623)
>
> Attachments: HDFS-2393-HDFS-1623.patch, HDFS-2393-HDFS-1623.patch
>
>
> HDFS-1973 will make the {{DFSClient}} take advantage of the annotation. This
> JIRA is to identify which methods can be marked as idempotent.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise readahead and drop-behind
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137718#comment-13137718 ] Eli Collins commented on HDFS-2465: --- bq. I don't think so. HDFS doesn't really do readahead except for filling up the TCP window I was thinking that filling up the dfs packet is essentially readahead since we'll read the whole 64kb (or whatever is configured) regardless of how much the client asks for but yea in this context that's not relevant (eg HBase won't issue such small reads). +1 to your latest patch. (Nit: add spaces in "1024*1024" no need to generate a new patch) > Add HDFS support for fadvise readahead and drop-behind > -- > > Key: HDFS-2465 > URL: https://issues.apache.org/jira/browse/HDFS-2465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2465.txt, hdfs-2465.txt > > > This is the HDFS side of HADOOP-7714. The initial implementation is heuristic > based and should be considered experimental, as discussed in the parent JIRA. > It should be off by default until better heuristics, APIs, and tuning > experience is developed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2418) Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
[
https://issues.apache.org/jira/browse/HDFS-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137712#comment-13137712
]
Hadoop QA commented on HDFS-2418:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501188/HDFS-2418.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1479//console
This message is automatically generated.
> Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
> -
>
> Key: HDFS-2418
> URL: https://issues.apache.org/jira/browse/HDFS-2418
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: HA branch (HDFS-1623)
>
> Attachments: HDFS-2418.patch, HDFS-2418.patch
>
>
> The {{ConfiguredFailoverProxyProvider}} will need to be amended to take
> advantage of the improvements to HA configuration introduced by HDFS-2231,
> once it's committed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2418) Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
[
https://issues.apache.org/jira/browse/HDFS-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-2418:
-
Resolution: Fixed
Fix Version/s: HA branch (HDFS-1623)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've just committed this.
> Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
> -
>
> Key: HDFS-2418
> URL: https://issues.apache.org/jira/browse/HDFS-2418
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: HA branch (HDFS-1623)
>
> Attachments: HDFS-2418.patch, HDFS-2418.patch
>
>
> The {{ConfiguredFailoverProxyProvider}} will need to be amended to take
> advantage of the improvements to HA configuration introduced by HDFS-2231,
> once it's committed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2418) Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
[
https://issues.apache.org/jira/browse/HDFS-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-2418:
-
Attachment: HDFS-2418.patch
Thanks a lot for the quick review, Eli. In this patch I've changed the name to
{{getHaNnRpcAddresses}}.
I'll commit this to the HA branch momentarily.
> Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
> -
>
> Key: HDFS-2418
> URL: https://issues.apache.org/jira/browse/HDFS-2418
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: HDFS-2418.patch, HDFS-2418.patch
>
>
> The {{ConfiguredFailoverProxyProvider}} will need to be amended to take
> advantage of the improvements to HA configuration introduced by HDFS-2231,
> once it's committed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137704#comment-13137704 ] Hudson commented on HDFS-2500: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1195 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1195/]) HDFS-2500. Avoid file system operations in BPOfferService thread while processing deletes. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190071 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137698#comment-13137698
]
Suresh Srinivas commented on HDFS-2298:
---
bq. Suresh, yes, I very much agree that this should be merged to 0.23. However,
since the reason is to keep the protocols compatible, then we need to first
merge HADOOP-7524 and HADOOP-7693 to 0.23. The version of this patch that works
without those would still leave 0.23 and trunk incompatible.
HADOOP-7524 made changes in ipc/rpc layer mechanism. I am not sure what you
mean by 0.23 and trunk would be incompatible without these.
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2418) Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
[
https://issues.apache.org/jira/browse/HDFS-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137692#comment-13137692
]
Eli Collins commented on HDFS-2418:
---
+1 lgtm
Nit: I'd include "RPC" in the method name of getHaNameNodeAddresses
> Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
> -
>
> Key: HDFS-2418
> URL: https://issues.apache.org/jira/browse/HDFS-2418
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: HDFS-2418.patch
>
>
> The {{ConfiguredFailoverProxyProvider}} will need to be amended to take
> advantage of the improvements to HA configuration introduced by HDFS-2231,
> once it's committed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
[ https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137690#comment-13137690 ] Hadoop QA commented on HDFS-2450: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501178/HDFS-2450-5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1478//console This message is automatically generated. > Only complete hostname is supported to access data via hdfs:// > -- > > Key: HDFS-2450 > URL: https://issues.apache.org/jira/browse/HDFS-2450 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Rajit Saha >Assignee: Daryn Sharp > Attachments: HDFS-2450-1.patch, HDFS-2450-2.patch, HDFS-2450-3.patch, > HDFS-2450-4.patch, HDFS-2450-5.patch, HDFS-2450.patch, IP vs. Hostname.pdf > > > If my complete hostname is host1.abc.xyz.com, only complete hostname must be > used to access data via hdfs:// > I am running following in .20.205 Client to get data from .20.205 NN (host1) > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp > copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/ > copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: > hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > $hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/ > copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: > hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > Only following is supported > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137688#comment-13137688 ] Hudson commented on HDFS-2500: -- Integrated in Hadoop-Common-0.23-Commit #87 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/87/]) HDFS-2500. Avoid file system operations in BPOfferService thread while processing deletes. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190072 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137689#comment-13137689 ] Hudson commented on HDFS-2500: -- Integrated in Hadoop-Common-trunk-Commit #1179 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1179/]) HDFS-2500. Avoid file system operations in BPOfferService thread while processing deletes. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190071 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137686#comment-13137686 ] Hudson commented on HDFS-2500: -- Integrated in Hadoop-Hdfs-trunk-Commit #1256 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1256/]) HDFS-2500. Avoid file system operations in BPOfferService thread while processing deletes. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190071 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137684#comment-13137684 ] Hudson commented on HDFS-2500: -- Integrated in Hadoop-Hdfs-0.23-Commit #87 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/87/]) HDFS-2500. Avoid file system operations in BPOfferService thread while processing deletes. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190072 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetAsyncDiskService.java > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise readahead and drop-behind
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137683#comment-13137683 ] Todd Lipcon commented on HDFS-2465: --- looks like the test-patch build ran before HADOOP-7753 was integrated in the snapshot build. I'll wait half an hour or so and retrigger this test-patch. Meanwhile, should be good to review. > Add HDFS support for fadvise readahead and drop-behind > -- > > Key: HDFS-2465 > URL: https://issues.apache.org/jira/browse/HDFS-2465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2465.txt, hdfs-2465.txt > > > This is the HDFS side of HADOOP-7714. The initial implementation is heuristic > based and should be considered experimental, as discussed in the parent JIRA. > It should be off by default until better heuristics, APIs, and tuning > experience is developed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2393) Mark appropriate methods of ClientProtocol with the idempotent annotation
[
https://issues.apache.org/jira/browse/HDFS-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137680#comment-13137680
]
Todd Lipcon commented on HDFS-2393:
---
+1
> Mark appropriate methods of ClientProtocol with the idempotent annotation
> -
>
> Key: HDFS-2393
> URL: https://issues.apache.org/jira/browse/HDFS-2393
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: HA branch (HDFS-1623)
>
> Attachments: HDFS-2393-HDFS-1623.patch
>
>
> HDFS-1973 will make the {{DFSClient}} take advantage of the annotation. This
> JIRA is to identify which methods can be marked as idempotent.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
[ https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137677#comment-13137677 ] Suresh Srinivas commented on HDFS-2450: --- +1 for the patch. > Only complete hostname is supported to access data via hdfs:// > -- > > Key: HDFS-2450 > URL: https://issues.apache.org/jira/browse/HDFS-2450 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Rajit Saha >Assignee: Daryn Sharp > Attachments: HDFS-2450-1.patch, HDFS-2450-2.patch, HDFS-2450-3.patch, > HDFS-2450-4.patch, HDFS-2450-5.patch, HDFS-2450.patch, IP vs. Hostname.pdf > > > If my complete hostname is host1.abc.xyz.com, only complete hostname must be > used to access data via hdfs:// > I am running following in .20.205 Client to get data from .20.205 NN (host1) > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp > copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/ > copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: > hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > $hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/ > copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: > hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > Only following is supported > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2500: -- Resolution: Fixed Fix Version/s: 0.23.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 23 and trunk. Thanks for the swift review, Eli! > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2500: -- Attachment: hdfs-2500-branch-0.23.txt branch-23 was slightly different, since AsyncDiskService's bpid parameter had gotten collapsed into an ExtendedBlock parameter instead of being passed separately. But the differences are trivial, so I'll commit without a re-review. > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2500-1.patch, hdfs-2500-branch-0.23.txt, > hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise readahead and drop-behind
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137673#comment-13137673 ] Hadoop QA commented on HDFS-2465: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501177/hdfs-2465.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1477//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1477//console This message is automatically generated. > Add HDFS support for fadvise readahead and drop-behind > -- > > Key: HDFS-2465 > URL: https://issues.apache.org/jira/browse/HDFS-2465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2465.txt, hdfs-2465.txt > > > This is the HDFS side of HADOOP-7714. The initial implementation is heuristic > based and should be considered experimental, as discussed in the parent JIRA. > It should be off by default until better heuristics, APIs, and tuning > experience is developed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
[
https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-2450:
--
Attachment: HDFS-2450-5.patch
Per suresh, renamed test resolver class, and upgraded
{{FileSystem#getCanonicalUri()}} visibility to protected.
> Only complete hostname is supported to access data via hdfs://
> --
>
> Key: HDFS-2450
> URL: https://issues.apache.org/jira/browse/HDFS-2450
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.20.205.0
>Reporter: Rajit Saha
>Assignee: Daryn Sharp
> Attachments: HDFS-2450-1.patch, HDFS-2450-2.patch, HDFS-2450-3.patch,
> HDFS-2450-4.patch, HDFS-2450-5.patch, HDFS-2450.patch, IP vs. Hostname.pdf
>
>
> If my complete hostname is host1.abc.xyz.com, only complete hostname must be
> used to access data via hdfs://
> I am running following in .20.205 Client to get data from .20.205 NN (host1)
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp
> copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal ... ]
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/
> copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected:
> hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal ... ]
> $hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/
> copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected:
> hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal ... ]
> Only following is supported
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead and drop-behind
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2465: -- Status: Patch Available (was: Open) > Add HDFS support for fadvise readahead and drop-behind > -- > > Key: HDFS-2465 > URL: https://issues.apache.org/jira/browse/HDFS-2465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2465.txt, hdfs-2465.txt > > > This is the HDFS side of HADOOP-7714. The initial implementation is heuristic > based and should be considered experimental, as discussed in the parent JIRA. > It should be off by default until better heuristics, APIs, and tuning > experience is developed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead and drop-behind
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2465: -- Component/s: performance > Add HDFS support for fadvise readahead and drop-behind > -- > > Key: HDFS-2465 > URL: https://issues.apache.org/jira/browse/HDFS-2465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2465.txt, hdfs-2465.txt > > > This is the HDFS side of HADOOP-7714. The initial implementation is heuristic > based and should be considered experimental, as discussed in the parent JIRA. > It should be off by default until better heuristics, APIs, and tuning > experience is developed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2465) Add HDFS support for fadvise readahead and drop-behind
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2465: -- Attachment: hdfs-2465.txt > Add HDFS support for fadvise readahead and drop-behind > -- > > Key: HDFS-2465 > URL: https://issues.apache.org/jira/browse/HDFS-2465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2465.txt, hdfs-2465.txt > > > This is the HDFS side of HADOOP-7714. The initial implementation is heuristic > based and should be considered experimental, as discussed in the parent JIRA. > It should be off by default until better heuristics, APIs, and tuning > experience is developed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2465) Add HDFS support for fadvise readahead and drop-behind
[ https://issues.apache.org/jira/browse/HDFS-2465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137667#comment-13137667 ] Todd Lipcon commented on HDFS-2465: --- bq. There isn't any config that enables syncBehindWrites in BlockerRecevier Woops! good catch. Lost that somewhere along the line. bq. Would it be more clear to express LONG_READ_THRESHOLD in terms of a multiple of the packet size or the IO file buffer size? Seems like the host readahead should match the implicit HDFS readahead. I don't think so. HDFS doesn't really do readahead except for filling up the TCP window. But submitting very large (multi-MB)readahead to the OS lets it schedule big sequential reads. So, I don't think there's any real correlation. To set this tunable, I'd probably do some mental math like: "well, I probably have 20 files being read at once, so 8M of readahead on each of those will end up being 160M of RAM used." Having to multiply by other factors is harder to follow. Fixed the other nits. New patch momentarily. > Add HDFS support for fadvise readahead and drop-behind > -- > > Key: HDFS-2465 > URL: https://issues.apache.org/jira/browse/HDFS-2465 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2465.txt > > > This is the HDFS side of HADOOP-7714. The initial implementation is heuristic > based and should be considered experimental, as discussed in the parent JIRA. > It should be off by default until better heuristics, APIs, and tuning > experience is developed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137656#comment-13137656 ] Todd Lipcon commented on HDFS-2500: --- failed tests are due to TestDfsOverAvroRpc timing out (already a JIRA filed for this). Will commit momentarily. Thanks for the review, Eli. > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2500-1.patch, hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137651#comment-13137651 ] Hadoop QA commented on HDFS-2500: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501160/hdfs-2500-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestBalancerBandwidth +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1475//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1475//console This message is automatically generated. > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2500-1.patch, hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2479) HDFS Client Data Types in Protocol Buffers
[ https://issues.apache.org/jira/browse/HDFS-2479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137648#comment-13137648 ] Sanjay Radia commented on HDFS-2479: Tests runs fine for me Running org.apache.hadoop.hdfs.TestFSInputChecker Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.542 sec Running org.apache.hadoop.tools.TestJMXGet Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.941 sec Running org.apache.hadoop.hdfs.TestSmallBlock Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.209 sec Running org.apache.hadoop.hdfs.TestListPathServlet Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.34 sec Running org.apache.hadoop.cli.TestHDFSCLI Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.918 sec Running org.apache.hadoop.security.TestRefreshUserMappings Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.788 sec Running org.apache.hadoop.hdfs.TestFileCreationClient Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.487 sec Running org.apache.hadoop.security.TestPermission Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.289 sec Running org.apache.hadoop.hdfs.TestSetrepDecreasing Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.13 sec TestHftpFileSystem a failure due to locking > HDFS Client Data Types in Protocol Buffers > -- > > Key: HDFS-2479 > URL: https://issues.apache.org/jira/browse/HDFS-2479 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: hdfs.proto, hdfs.proto, hdfs.proto, > pbClientTypes1.patch, pbClientTypes2.patch, pbClientTypes3.patch, > pbClientTypes4.patch, pbClientTypes5.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2507) HA: Allow saveNamespace operations to be canceled
[ https://issues.apache.org/jira/browse/HDFS-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137644#comment-13137644 ] Hudson commented on HDFS-2507: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1193 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1193/]) HDFS-2507. Allow saveNamespace operations to be canceled. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190060 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceCancelledException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java > HA: Allow saveNamespace operations to be canceled > - > > Key: HDFS-2507 > URL: https://issues.apache.org/jira/browse/HDFS-2507 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: HA branch (HDFS-1623), 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: hdfs-2507.txt > > > For HDFS-2291 (checkpointing in HA setup) we need to allow the StandbyNode to > handle a failover while it may be in the middle of saving a checkpoint. This > implies that checkpoints need to be abortable. > This JIRA is to implement this functionality. Even though this is only needed > for HA, I think we should do the infrastructure in trunk to simplify future > merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2418) Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
[
https://issues.apache.org/jira/browse/HDFS-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137624#comment-13137624
]
Hadoop QA commented on HDFS-2418:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501165/HDFS-2418.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1476//console
This message is automatically generated.
> Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
> -
>
> Key: HDFS-2418
> URL: https://issues.apache.org/jira/browse/HDFS-2418
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: HDFS-2418.patch
>
>
> The {{ConfiguredFailoverProxyProvider}} will need to be amended to take
> advantage of the improvements to HA configuration introduced by HDFS-2231,
> once it's committed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2507) HA: Allow saveNamespace operations to be canceled
[ https://issues.apache.org/jira/browse/HDFS-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137618#comment-13137618 ] Hudson commented on HDFS-2507: -- Integrated in Hadoop-Hdfs-trunk-Commit #1254 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1254/]) HDFS-2507. Allow saveNamespace operations to be canceled. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190060 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceCancelledException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java > HA: Allow saveNamespace operations to be canceled > - > > Key: HDFS-2507 > URL: https://issues.apache.org/jira/browse/HDFS-2507 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: HA branch (HDFS-1623), 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: hdfs-2507.txt > > > For HDFS-2291 (checkpointing in HA setup) we need to allow the StandbyNode to > handle a failover while it may be in the middle of saving a checkpoint. This > implies that checkpoints need to be abortable. > This JIRA is to implement this functionality. Even though this is only needed > for HA, I think we should do the infrastructure in trunk to simplify future > merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2507) HA: Allow saveNamespace operations to be canceled
[ https://issues.apache.org/jira/browse/HDFS-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137619#comment-13137619 ] Hudson commented on HDFS-2507: -- Integrated in Hadoop-Common-trunk-Commit #1177 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1177/]) HDFS-2507. Allow saveNamespace operations to be canceled. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190060 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceCancelledException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/MD5FileUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java > HA: Allow saveNamespace operations to be canceled > - > > Key: HDFS-2507 > URL: https://issues.apache.org/jira/browse/HDFS-2507 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: HA branch (HDFS-1623), 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: hdfs-2507.txt > > > For HDFS-2291 (checkpointing in HA setup) we need to allow the StandbyNode to > handle a failover while it may be in the middle of saving a checkpoint. This > implies that checkpoints need to be abortable. > This JIRA is to implement this functionality. Even though this is only needed > for HA, I think we should do the infrastructure in trunk to simplify future > merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2507) HA: Allow saveNamespace operations to be canceled
[ https://issues.apache.org/jira/browse/HDFS-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2507: -- Resolution: Fixed Fix Version/s: 0.24.0 HA branch (HDFS-1623) Target Version/s: HA branch (HDFS-1623), 0.24.0 (was: 0.24.0, HA branch (HDFS-1623)) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to HA and 0.24. I wasn't able to commit to 0.23 since HDFS-2223 is missing, there. We may have to backport both 2223 and this JIRA when we merge HA. > HA: Allow saveNamespace operations to be canceled > - > > Key: HDFS-2507 > URL: https://issues.apache.org/jira/browse/HDFS-2507 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: HA branch (HDFS-1623), 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623), 0.24.0 > > Attachments: hdfs-2507.txt > > > For HDFS-2291 (checkpointing in HA setup) we need to allow the StandbyNode to > handle a failover while it may be in the middle of saving a checkpoint. This > implies that checkpoints need to be abortable. > This JIRA is to implement this functionality. Even though this is only needed > for HA, I think we should do the infrastructure in trunk to simplify future > merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2418) Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
[
https://issues.apache.org/jira/browse/HDFS-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-2418:
-
Attachment: HDFS-2418.patch
Here's a patch which addresses the issue. Note that this patch also fixes a bug
in {{DFSUtil.getAddresses(...)}} wherein addresses would only be successfully
retrieved if both HA and federation were enabled, but not if only one of these
features was enabled.
> Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
> -
>
> Key: HDFS-2418
> URL: https://issues.apache.org/jira/browse/HDFS-2418
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: HDFS-2418.patch
>
>
> The {{ConfiguredFailoverProxyProvider}} will need to be amended to take
> advantage of the improvements to HA configuration introduced by HDFS-2231,
> once it's committed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2418) Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
[
https://issues.apache.org/jira/browse/HDFS-2418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-2418:
-
Status: Patch Available (was: Open)
> Change ConfiguredFailoverProxyProvider to take advantage of HDFS-2231
> -
>
> Key: HDFS-2418
> URL: https://issues.apache.org/jira/browse/HDFS-2418
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs client
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: HDFS-2418.patch
>
>
> The {{ConfiguredFailoverProxyProvider}} will need to be amended to take
> advantage of the improvements to HA configuration introduced by HDFS-2231,
> once it's committed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2507) HA: Allow saveNamespace operations to be canceled
[ https://issues.apache.org/jira/browse/HDFS-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137595#comment-13137595 ] Eli Collins commented on HDFS-2507: --- +1 looks great. Nit: no need to pass null to dirIterable in the test > HA: Allow saveNamespace operations to be canceled > - > > Key: HDFS-2507 > URL: https://issues.apache.org/jira/browse/HDFS-2507 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: HA branch (HDFS-1623), 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2507.txt > > > For HDFS-2291 (checkpointing in HA setup) we need to allow the StandbyNode to > handle a failover while it may be in the middle of saving a checkpoint. This > implies that checkpoints need to be abortable. > This JIRA is to implement this functionality. Even though this is only needed > for HA, I think we should do the infrastructure in trunk to simplify future > merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2432) webhdfs setreplication api should return a 403 when called on a directory
[
https://issues.apache.org/jira/browse/HDFS-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137581#comment-13137581
]
Jitendra Nath Pandey commented on HDFS-2432:
+1 lgtm.
> webhdfs setreplication api should return a 403 when called on a directory
> -
>
> Key: HDFS-2432
> URL: https://issues.apache.org/jira/browse/HDFS-2432
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2432_20111026.patch, h2432_20111026_0.20s.patch,
> h2432_20111027.patch, h2432_20111027_0.20s.patch
>
>
> Currently the set replication api on a directory leads to a 200.
> Request URI
> http://NN:50070/webhdfs/tmp/webhdfs_data/dir_replication_tests?op=SETREPLICATION&replication=5
> Request Method: PUT
> Status Line: HTTP/1.1 200 OK
> Response Content: {"boolean":false}
> Since we can determine that this call did not succeed (boolean=false) we
> should rather just return a 403
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2500) Avoid file system operations in BPOfferService thread while processing deletes
[ https://issues.apache.org/jira/browse/HDFS-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2500: -- Attachment: hdfs-2500-1.patch Looks great. +1 pending hudson. Updated patch attach is rebased on trunk. > Avoid file system operations in BPOfferService thread while processing deletes > -- > > Key: HDFS-2500 > URL: https://issues.apache.org/jira/browse/HDFS-2500 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2500-1.patch, hdfs-2500.txt > > > While running a workload with concurrent writes and deletes, I saw a lot of > NotReplicatedYetExceptions being thrown due to late arrivals of blockReceived > reports from the DN. Looking at the DN logs, I found that the blockReceived > message was being delayed as much as 15 seconds because the OfferService > thread was blocked on file system operations processing deletes. We > previously moved the deletions to another thread, but it still accesses the > file system to determine the block length in the main thread. On a heavily > loaded system this can take a long time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137541#comment-13137541
]
Hudson commented on HDFS-2298:
--
Integrated in Hadoop-Mapreduce-0.23-Commit #77 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/77/])
svn merge -c 1177757 from trunk for HDFS-2298.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189983
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
[ https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137524#comment-13137524 ] Hadoop QA commented on HDFS-2450: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12501154/HDFS-2450-4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1474//console This message is automatically generated. > Only complete hostname is supported to access data via hdfs:// > -- > > Key: HDFS-2450 > URL: https://issues.apache.org/jira/browse/HDFS-2450 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Rajit Saha >Assignee: Daryn Sharp > Attachments: HDFS-2450-1.patch, HDFS-2450-2.patch, HDFS-2450-3.patch, > HDFS-2450-4.patch, HDFS-2450.patch, IP vs. Hostname.pdf > > > If my complete hostname is host1.abc.xyz.com, only complete hostname must be > used to access data via hdfs:// > I am running following in .20.205 Client to get data from .20.205 NN (host1) > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp > copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/ > copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: > hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > $hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/ > copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected: > hdfs://host1.abc.xyz.com > Usage: java FsShell [-copyFromLocal ... ] > Only following is supported > $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2366) webhdfs throws a npe when ugi is null from getDelegationToken
[ https://issues.apache.org/jira/browse/HDFS-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2366: - Issue Type: Sub-task (was: Bug) Parent: HDFS-2316 > webhdfs throws a npe when ugi is null from getDelegationToken > - > > Key: HDFS-2366 > URL: https://issues.apache.org/jira/browse/HDFS-2366 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.20.205.0 >Reporter: Arpit Gupta >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.20.205.0, 0.20.206.0, 0.23.0, 0.24.0 > > Attachments: h2366_20110926.patch, h2366_20110926_0.20s.patch, > h2366_20110926_0.20s_b.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2450) Only complete hostname is supported to access data via hdfs://
[
https://issues.apache.org/jira/browse/HDFS-2450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-2450:
--
Attachment: HDFS-2450-4.patch
Reduced public visibility of {{FileSystem#getCanonicalUri()}}.
Shuffled the tests around.
> Only complete hostname is supported to access data via hdfs://
> --
>
> Key: HDFS-2450
> URL: https://issues.apache.org/jira/browse/HDFS-2450
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.20.205.0
>Reporter: Rajit Saha
>Assignee: Daryn Sharp
> Attachments: HDFS-2450-1.patch, HDFS-2450-2.patch, HDFS-2450-3.patch,
> HDFS-2450-4.patch, HDFS-2450.patch, IP vs. Hostname.pdf
>
>
> If my complete hostname is host1.abc.xyz.com, only complete hostname must be
> used to access data via hdfs://
> I am running following in .20.205 Client to get data from .20.205 NN (host1)
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1/tmp
> copyFromLocal: Wrong FS: hdfs://host1/tmp, expected: hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal ... ]
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc/tmp/
> copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected:
> hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal ... ]
> $hadoop dfs -copyFromLocal /etc/passwd hftp://host1.abc.xyz/tmp/
> copyFromLocal: Wrong FS: hdfs://host1.blue/tmp/1, expected:
> hdfs://host1.abc.xyz.com
> Usage: java FsShell [-copyFromLocal ... ]
> Only following is supported
> $hadoop dfs -copyFromLocal /etc/passwd hdfs://host1.abc.xyz.com/tmp/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2355) Federation: enable using the same configuration file across all the nodes in the cluster.
[ https://issues.apache.org/jira/browse/HDFS-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137498#comment-13137498 ] Hudson commented on HDFS-2355: -- Integrated in Hadoop-Mapreduce-0.23-Commit #76 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/76/]) svn merge -c 1177100 from trunk for HDFS-2355. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189979 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestMulitipleNNDataBlockScanner.java > Federation: enable using the same configuration file across all the nodes in > the cluster. > - > > Key: HDFS-2355 > URL: https://issues.apache.org/jira/browse/HDFS-2355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2355.txt, HDFS-2355.txt > > > In federation the configuration files xml include another configuration xml. > This configuration xml is empty for datanodes and defines nameservice ID for > namenodes, secondary and backup nodes. The need for xml include can be > eliminated by namenodes determining the nameservice ID by matching its local > address with a configured address (such as namenode rpc address) and on match > using the nameservice ID of the corresponding configuration parameter. > For more details about federation configuration see > https://issues.apache.org/jira/browse/HDFS-1689?focusedCommentId=13001749&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13001749 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2403) The renewer in NamenodeWebHdfsMethods.generateDelegationToken(..) is not used
[ https://issues.apache.org/jira/browse/HDFS-2403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2403: - Issue Type: Sub-task (was: Bug) Parent: HDFS-2316 > The renewer in NamenodeWebHdfsMethods.generateDelegationToken(..) is not used > - > > Key: HDFS-2403 > URL: https://issues.apache.org/jira/browse/HDFS-2403 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.20.205.0, 0.23.0, 0.24.0 > > Attachments: h2403_20111005.patch, h2403_20111005_0.20.patch > > > Below are some suggestions from Suresh. > # renewer not used in #generateDelegationToken > # put() does not use InputStream in and should not throw URISyntaxException > # post() does not use InputStream in and should not throw URISyntaxException > # get() should not throw URISyntaxException -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2371) Refactor BlockSender.java for better readability
[ https://issues.apache.org/jira/browse/HDFS-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137499#comment-13137499 ] Hudson commented on HDFS-2371: -- Integrated in Hadoop-Mapreduce-0.23-Commit #76 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/76/]) svn merge -c 1177161 from trunk for HDFS-2371. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189981 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java > Refactor BlockSender.java for better readability > > > Key: HDFS-2371 > URL: https://issues.apache.org/jira/browse/HDFS-2371 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2371.txt, HDFS-2371.txt, HDFS-2371.txt > > > BlockSender.java is hard to read and understand. I propose refactoring it for > better readability -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137494#comment-13137494
]
Tsz Wo (Nicholas), SZE commented on HDFS-2298:
--
Hi Doug,
Suresh has suggested to merge this to 0.23 on the hdfs-dev mailing list. Below
is his justification.
> HADOOP-2298 TestDFSOverAvroRpc fails
> Justification: This makes a protocol method name change. Required for
> compatibility of 0.23 with future releases.
> Risk: low
Do you agree? Or we may revert the 0.23 commit.
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137485#comment-13137485
]
Hudson commented on HDFS-2298:
--
Integrated in Hadoop-Hdfs-0.23-Commit #81 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/81/])
svn merge -c 1177757 from trunk for HDFS-2298.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189983
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137489#comment-13137489
]
Hudson commented on HDFS-2298:
--
Integrated in Hadoop-Common-0.23-Commit #81 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/81/])
svn merge -c 1177757 from trunk for HDFS-2298.
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189983
Files :
* /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137481#comment-13137481
]
Doug Cutting commented on HDFS-2298:
Nicholas, so you've elected to merge this without HADOOP-7524 and HADOOP-7693?
That will synchronize the protocols but won't fix test failures in 0.23. Is
that your intent? It would be much better to present such a proposal before
committing.
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2298) TestDfsOverAvroRpc is failing on trunk
[
https://issues.apache.org/jira/browse/HDFS-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2298:
-
Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0)
Fix Version/s: 0.23.0
Merged to 0.23.
> TestDfsOverAvroRpc is failing on trunk
> --
>
> Key: HDFS-2298
> URL: https://issues.apache.org/jira/browse/HDFS-2298
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Aaron T. Myers
>Assignee: Doug Cutting
> Fix For: 0.23.0, 0.24.0
>
> Attachments: HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch,
> HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch, HDFS-2298.patch
>
>
> The relevant bit of the error:
> {noformat}
> ---
> Test set: org.apache.hadoop.hdfs.TestDfsOverAvroRpc
> ---
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.486 sec <<<
> FAILURE!
> testWorkingDirectory(org.apache.hadoop.hdfs.TestDfsOverAvroRpc) Time
> elapsed: 1.424 sec <<< ERROR!
> org.apache.avro.AvroTypeException: Two methods with same name: delete
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2371) Refactor BlockSender.java for better readability
[ https://issues.apache.org/jira/browse/HDFS-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137459#comment-13137459 ] Hudson commented on HDFS-2371: -- Integrated in Hadoop-Common-0.23-Commit #80 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/80/]) svn merge -c 1177161 from trunk for HDFS-2371. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189981 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java > Refactor BlockSender.java for better readability > > > Key: HDFS-2371 > URL: https://issues.apache.org/jira/browse/HDFS-2371 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2371.txt, HDFS-2371.txt, HDFS-2371.txt > > > BlockSender.java is hard to read and understand. I propose refactoring it for > better readability -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2371) Refactor BlockSender.java for better readability
[ https://issues.apache.org/jira/browse/HDFS-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137456#comment-13137456 ] Hudson commented on HDFS-2371: -- Integrated in Hadoop-Hdfs-0.23-Commit #80 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/80/]) svn merge -c 1177161 from trunk for HDFS-2371. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189981 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceScanner.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java > Refactor BlockSender.java for better readability > > > Key: HDFS-2371 > URL: https://issues.apache.org/jira/browse/HDFS-2371 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2371.txt, HDFS-2371.txt, HDFS-2371.txt > > > BlockSender.java is hard to read and understand. I propose refactoring it for > better readability -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2355) Federation: enable using the same configuration file across all the nodes in the cluster.
[ https://issues.apache.org/jira/browse/HDFS-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137458#comment-13137458 ] Hudson commented on HDFS-2355: -- Integrated in Hadoop-Common-0.23-Commit #80 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/80/]) svn merge -c 1177100 from trunk for HDFS-2355. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189979 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestMulitipleNNDataBlockScanner.java > Federation: enable using the same configuration file across all the nodes in > the cluster. > - > > Key: HDFS-2355 > URL: https://issues.apache.org/jira/browse/HDFS-2355 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0, 0.24.0 > > Attachments: HDFS-2355.txt, HDFS-2355.txt > > > In federation the configuration files xml include another configuration xml. > This configuration xml is empty for datanodes and defines nameservice ID for > namenodes, secondary and backup nodes. The need for xml include can be > eliminated by namenodes determining the nameservice ID by matching its local > address with a configured address (such as namenode rpc address) and on match > using the nameservice ID of the corresponding configuration parameter. > For more details about federation configuration see > https://issues.apache.org/jira/browse/HDFS-1689?focusedCommentId=13001749&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13001749 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
