[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuweimin updated HDFS-245: --- Attachment: symlink-0.20.0.patch this a copy of symLink14.patch for hadoop released version 0.20.0 usage: put symlink-0.20.0.patch into $HADOOP_HOME cd $HADOOP_HOME patch -p0 < symlink-0.20.0.patch ant jar mv hadoop-0.20.0-core.jar hadoop-0.20.0-core.jar.bak cp build/hadoop-0.20.1-dev-core.jar restart the hadoop system at last > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, HADOOP-4044-strawman.patch, > symlink-0.20.0.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-265) Revisit append
[ https://issues.apache.org/jira/browse/HDFS-265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-265: Attachment: AppendTestPlan.html Normal pipeline tests information is updated per HDFS-608 > Revisit append > -- > > Key: HDFS-265 > URL: https://issues.apache.org/jira/browse/HDFS-265 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: appendDesign.pdf, appendDesign.pdf, appendDesign1.pdf, > appendDesign2.pdf, AppendSpec.pdf, AppendTestPlan.html, AppendTestPlan.html, > AppendTestPlan.html, AppendTestPlan.html, AppendTestPlan.html, > AppendTestPlan.html, AppendTestPlan.html, TestPlanAppend.html > > > HADOOP-1700 and related issues have put a lot of efforts to provide the first > implementation of append. However, append is such a complex feature. It turns > out that there are issues that were initially seemed trivial but needs a > careful design. This jira revisits append, aiming for a design and > implementation supporting a semantics that are acceptable to its users. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-616) Create functional tests for new design of the block report
Create functional tests for new design of the block report -- Key: HDFS-616 URL: https://issues.apache.org/jira/browse/HDFS-616 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-598) Eclipse launch task for HDFS
[ https://issues.apache.org/jira/browse/HDFS-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755020#action_12755020 ] Tom White commented on HDFS-598: This looks useful. I managed to run a specific test, but when I run AllTests I get an error saying "The input element of the launch configuration does not exist". > Eclipse launch task for HDFS > > > Key: HDFS-598 > URL: https://issues.apache.org/jira/browse/HDFS-598 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build > Environment: Eclipse 3.5 >Reporter: Eli Collins >Assignee: Eli Collins >Priority: Trivial > Attachments: hdfs-598.patch > > > Porting HDFS launch task from HADOOP-5911. See MAPREDUCE-905. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-598) Eclipse launch task for HDFS
[ https://issues.apache.org/jira/browse/HDFS-598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White reassigned HDFS-598: -- Assignee: Eli Collins > Eclipse launch task for HDFS > > > Key: HDFS-598 > URL: https://issues.apache.org/jira/browse/HDFS-598 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build > Environment: Eclipse 3.5 >Reporter: Eli Collins >Assignee: Eli Collins >Priority: Trivial > Attachments: hdfs-598.patch > > > Porting HDFS launch task from HADOOP-5911. See MAPREDUCE-905. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-592) Allow client to get a new generation stamp from NameNode
[ https://issues.apache.org/jira/browse/HDFS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755073#action_12755073 ] Hairong Kuang commented on HDFS-592: > The initial pipeline setup for append should get its access token from the > append() call (like how it is done today). There is an undetermined delay from the time append is called to the time the initial pipeline is set. A pipeline is set up when a client needs to send out the first packet. As we discussed at the design time, an access token has to be fetched right before a pipeline is set up. This behavior is parallel to what we do with file creation. When a client issues create, NN does not return any access token. But before a pipeline is set up, the client calls addBlock to get a new block and an access token. > Allow client to get a new generation stamp from NameNode > > > Key: HDFS-592 > URL: https://issues.apache.org/jira/browse/HDFS-592 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: newGS.patch, newGS1.patch > > > This issue aims to add an API to ClientProtocol that fetches a new > generation stamp and an access token from NameNode to support append or > pipeline recovery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-604) Block report processing for append
[ https://issues.apache.org/jira/browse/HDFS-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-604: - Attachment: BlockReportNN.patch > 1. When a block is under construction, ... NN should only allow to add new > replicas. Done. > 2. NN should not add storedBlock to toCorrupt list if storedBlock' GS is not > the same as that of the reported replica. Otherwise a wrong GS of the replica > will get deleted. This is tricky. I have been back and forth with this. There is a difference between corrupt and invalid replicas. _Invalid replica_ is the one that corresponds to a block that is not known to the name-node and therefore the replica can be safely removed from the data-node right away. _Corrupt replicas_ correspond to blocks that exist on the name-name, but the state of the replica is different from what the name-node expects. This should include both size and GS. A corrupt replica cannot be immediately removed from the data-node. First the name-node should trigger replication, and remove it only when the block is fully replicated. All this logic would work fine with my patch if the issue with keys HDFS-512 was resolved. I think we really need it. For now I included temporarily invalidation of replicas with wrong generation stamp. I plan to remove it and fix corrupt replica processing, which should remember corrupt replicas with wrong GS reported by data-nodes. Will address the safe-mode issue in a separate issue. > Block report processing for append > -- > > Key: HDFS-604 > URL: https://issues.apache.org/jira/browse/HDFS-604 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko > Fix For: Append Branch > > Attachments: BlockReportNN.patch, BlockReportNN.patch, > BlockReportNN.patch > > > Implement new block report processing on the name-node as stated in the > append design and HDFS-576. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755048#action_12755048 ] Todd Lipcon commented on HDFS-516: -- I haven't had a chance to look over the patch as of yet, but I have one concern: Is there a plan for deprecation in the event that HDFS itself achieves similar performance? I think having an entirely separate FS implementation that only differs in performance is not a good idea longterm. Using this contrib project as an experimentation ground sounds great, but I think long term we should focus on improving DistributedFileSystem's performance itself, and not bifurcate the code into the "fast version that we dont really support because it's contrib" and "slow version that we do". I'll try to find a chance to look over the patch soon, but in the meantime do you have any thoughts on the above? > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090912.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755033#action_12755033 ] Raghu Angadi commented on HDFS-516: --- Hi Jay, will go through the patch. I hope a few others get a chance to look at it as well. Since it is contrib, it certainly makes it easier to include in trunk. I am not sure about 0.21 timeline. > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090912.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-615) TestLargeDirectoryDelete fails with NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755056#action_12755056
]
Konstantin Boudnik commented on HDFS-615:
-
It seems to me that what {{MiniDFSCluster.waitActive()}} does contradicts to
the semantic of the call. It checks if the {{namenode}} reference is null and
then simply returns. Apparently the very first call which needs {{namenode}}
reference dumps NPE immediately.
Shall it be fixed along the lines of actually waiting for the namenode to start?
> TestLargeDirectoryDelete fails with NullPointerException
>
>
> Key: HDFS-615
> URL: https://issues.apache.org/jira/browse/HDFS-615
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.21.0
> Environment: 64-bit debian 5, 64-bit sun java6, running in a single
> processor VM.
>Reporter: Eli Collins
>Priority: Minor
>
> I've hit the following failure two out of two times running "ant test" at rev
> 813587. This test doesn't appear to be failing on hudson. All other tests
> passed except TestHDFSFileSystemContract which timed out, so perhaps there's
> a race due to the test executing slowly.
> [junit] Running
> org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete
> [junit] Exception in thread "Thread-30148" java.lang.NullPointerException
> [junit] at
> org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter.getNamesystem(NameNodeAdapter.java:32)
> [junit] at
> org.apache.hadoop.hdfs.MiniDFSCluster.getNamesystem(MiniDFSCluster.java:522)
> [junit] at
> org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete.getBlockCount(TestLargeDirectoryDelete.java:75)
> [junit] at
> org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete.access$000(TestLargeDirectoryDelete.java:38)
> [junit] at
> org.apache.hadoop.hdfs.server.namenode.TestLargeDirectoryDelete$1.run(TestLargeDirectoryDelete.java:90)
> [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 94.264 sec
> No failures
> or errors?
> public static FSNamesystem getNamesystem(NameNode namenode) {
> return namenode.getNamesystem();
> <===
> }
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-592) Allow client to get a new generation stamp from NameNode
[ https://issues.apache.org/jira/browse/HDFS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755083#action_12755083 ] Kan Zhang commented on HDFS-592: sorry, I forgot pipeline will not be created right after append() call. In that case, the client can use pipelineRecovery() call to get an access token since it already holds the lease. > Allow client to get a new generation stamp from NameNode > > > Key: HDFS-592 > URL: https://issues.apache.org/jira/browse/HDFS-592 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: newGS.patch, newGS1.patch > > > This issue aims to add an API to ClientProtocol that fetches a new > generation stamp and an access token from NameNode to support append or > pipeline recovery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-254) Add more unit test for HDFS symlinks
[ https://issues.apache.org/jira/browse/HDFS-254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HDFS-254: Assignee: Eli Collins (was: Henry Robinson) > Add more unit test for HDFS symlinks > > > Key: HDFS-254 > URL: https://issues.apache.org/jira/browse/HDFS-254 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: dhruba borthakur >Assignee: Eli Collins > > HADOOP-4044 introduces symbolic links in HDFS. That patch has some unit tests > associated with it. The aim of this JIRA is to add more unit tests (including > negative tests): > deleting a symlink that points to a file > deleting a symlink that points to a directory > symlink to a symlink > any test cases with relative links > delete linked to file, but not link, access link > delete linked to file, recreate the same file, access link > links to other filesystems > archives containing links > rename a file to an existing symlink > rename a symlink to an existing file > setTimes and setReplication cases should check the result on both the symlink > and linked to file > tests involving lack of permissions to create symlinks > tests that try to createSymlinks with filesystems that don't support them > ... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HDFS-245: Assignee: Eli Collins (was: dhruba borthakur) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, HADOOP-4044-strawman.patch, > symlink-0.20.0.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-255) Create a user-guide for describing HDFS symlinks
[ https://issues.apache.org/jira/browse/HDFS-255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HDFS-255: Assignee: Eli Collins (was: Henry Robinson) > Create a user-guide for describing HDFS symlinks > > > Key: HDFS-255 > URL: https://issues.apache.org/jira/browse/HDFS-255 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: dhruba borthakur >Assignee: Eli Collins > > HADOOP-4044 introduces symlinks in HDFS. We need a user-guide (aka man-page) > that describes the semantics of this new construct. > . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-604) Block report processing for append
[
https://issues.apache.org/jira/browse/HDFS-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755089#action_12755089
]
Konstantin Shvachko commented on HDFS-604:
--
test-core and test-patch succesfull
{code}
.[exec] There appear to be 150 release audit warnings before the patch and
150 release audit warnings after applying the patch.
[exec] +1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
==
[exec]
==
[exec] Finished build.
[exec]
==
[exec]
==
{code}
> Block report processing for append
> --
>
> Key: HDFS-604
> URL: https://issues.apache.org/jira/browse/HDFS-604
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Affects Versions: Append Branch
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: Append Branch
>
> Attachments: BlockReportNN.patch, BlockReportNN.patch,
> BlockReportNN.patch
>
>
> Implement new block report processing on the name-node as stated in the
> append design and HDFS-576.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-617) Support for non-recursive create() in HDFS
Support for non-recursive create() in HDFS -- Key: HDFS-617 URL: https://issues.apache.org/jira/browse/HDFS-617 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kan Zhang Assignee: Kan Zhang HADOOP-4952 calls for a create call that doesn't automatically create missing parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755099#action_12755099 ] Eli Collins commented on HDFS-245: -- Discussed with Dhruba. I'll start putting together a design doc to cover the above questions/comments which deserve some more discussion. Will also start on a test plan and user guide. I rebased the latest diffs against trunk and have patches for hdfs, common and mapreduce. I'll file jiras for the last two and update each with the relevant patch. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, HADOOP-4044-strawman.patch, > symlink-0.20.0.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-618) Support for non-recursive mkdir in HDFS
Support for non-recursive mkdir in HDFS --- Key: HDFS-618 URL: https://issues.apache.org/jira/browse/HDFS-618 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kan Zhang Assignee: Kan Zhang Existing mkdirs call automatically creates missing parent directories. HADOOP-4952 call for a mkdir call that doesn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-589) Change block write protocol to support pipeline recovery
[ https://issues.apache.org/jira/browse/HDFS-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755097#action_12755097 ] Hairong Kuang commented on HDFS-589: Thank Suresh and Nicholas for your review comments and suggestions. > DataStreamer for append has if (freeInLastBlock > blockSize) check. It will > never be true right? This seems to be a bug in the legacy code, the check should be freeInLastBlock == blockSize. Regarding BlockConstructionStage.valueOf, all your suggestions are very constructive. I am thinking to change it to be public BlockConstructionStage getRecoveryStage() which always returns a recovery enumerate. > Change block write protocol to support pipeline recovery > > > Key: HDFS-589 > URL: https://issues.apache.org/jira/browse/HDFS-589 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: opWriteProtocol.patch > > > Current block write operation's header has the following fields: > blockId blockGS pipelineSize isRecovery clientName hasSource source > #datanodesInDownStreamPipeline downstreamDatanodes > I'd like to change the header to be > blockId blockGS pipelineSize clientName flags blockMinLen blockMaxLen newGS > hasSource source #datanodesInDownStreamPipeline downstreamDatanodes > With this protocol change, pipeline recovery will be performed when a mew > pipeline is set up. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-604) Block report processing for append
[ https://issues.apache.org/jira/browse/HDFS-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved HDFS-604. -- Resolution: Fixed Hadoop Flags: [Reviewed] I just committed this. > Block report processing for append > -- > > Key: HDFS-604 > URL: https://issues.apache.org/jira/browse/HDFS-604 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko > Fix For: Append Branch > > Attachments: BlockReportNN.patch, BlockReportNN.patch, > BlockReportNN.patch > > > Implement new block report processing on the name-node as stated in the > append design and HDFS-576. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Attachment: h617-01.patch > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755117#action_12755117 ] dhruba borthakur commented on HDFS-385: --- I will commit this in a day. > Design a pluggable interface to place replicas of blocks in HDFS > > > Key: HDFS-385 > URL: https://issues.apache.org/jira/browse/HDFS-385 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: BlockPlacementPluggable.txt, > BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, > BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, > BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, > BlockPlacementPluggable7.txt > > > The current HDFS code typically places one replica on local rack, the second > replica on remote random rack and the third replica on a random node of that > remote rack. This algorithm is baked in the NameNode's code. It would be nice > to make the block placement algorithm a pluggable interface. This will allow > experimentation of different placement algorithms based on workloads, > availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755120#action_12755120 ] Jay Booth commented on HDFS-516: Hey Todd, in short, I agree, we should be looking at moving performance improvements over to the main FS implementation. Right now, my version doesn't support user permissions or checksumming. I'd say it makes sense to keep it in contrib as a sandbox for now, and work towards full compatibility with the main DFS implementation at which point we could consider swapping in the new reading subsystem? User permissioning would require some model changes but should be workable, checksumming probably won't be too bad if I read the code right. So, I suppose keep it in contrib as a sandbox initially with an explicit goal of moving it over to DFS when it reaches compatibility? It doesn't really lend itself to moving over piecemeal, as it has several components which all pretty much need each other. However, it's pretty well integrated with the DFS API and only replaces one method on the filesystem class. > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090912.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-516) Low Latency distributed reads
[ https://issues.apache.org/jira/browse/HDFS-516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755121#action_12755121 ] Todd Lipcon commented on HDFS-516: -- Jay: that sounds good to me. Let's explicitly mark the API as "Experimental" and "Likely to disappear in a future release with little or no warning - use at your own risk" :) I think especially as security and authentication begin to take form in the next couple months it will be a headache to try to maintain this code. If branching in SVN weren't such a pain, I'd suggest we maintain this in a separate branch that didn't go into releases... c'est la vie ;-) > Low Latency distributed reads > - > > Key: HDFS-516 > URL: https://issues.apache.org/jira/browse/HDFS-516 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jay Booth >Priority: Minor > Attachments: hdfs-516-20090912.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > I created a method for low latency random reads using NIO on the server side > and simulated OS paging with LRU caching and lookahead on the client side. > Some applications could include lucene searching (term->doc and doc->offset > mappings are likely to be in local cache, thus much faster than nutch's > current FsDirectory impl and binary search through record files (bytes at > 1/2, 1/4, 1/8 marks are likely to be cached) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Status: Patch Available (was: Open) > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755133#action_12755133 ] Kan Zhang commented on HDFS-617: Atteched a patch that 1. added a boolean parameter to the create() RPC call to indicate whether missing parent directories should be created. 2. added a createNonCursive() method to DistributedFileSystem interface, which doesn't create missing parent directories (complements existing create() method that does). > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-592) Allow client to get a new generation stamp from NameNode
[ https://issues.apache.org/jira/browse/HDFS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755136#action_12755136 ] Hairong Kuang commented on HDFS-592: > the client can use pipelineRecovery() call to get an access token since it > already holds the lease. My main point is that I still want to keep the method name as it is in the patch or a short form like getNewStampAndToken(). Getting a new GS and access token is just one small step towards pipeline setup/recovery. The name pipelineRecovery() is too vague. > Allow client to get a new generation stamp from NameNode > > > Key: HDFS-592 > URL: https://issues.apache.org/jira/browse/HDFS-592 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: newGS.patch, newGS1.patch > > > This issue aims to add an API to ClientProtocol that fetches a new > generation stamp and an access token from NameNode to support append or > pipeline recovery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-254) Add more unit test for HDFS symlinks
[ https://issues.apache.org/jira/browse/HDFS-254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755147#action_12755147 ] Konstantin Boudnik commented on HDFS-254: - A couple of points for the negative tests (might want to use fault injection framework HDFS-435 for this) - throw DiskErrorException during symlink removal - throw DiskOutOfSpaceException during symlink creation - creating looped symlinks > Add more unit test for HDFS symlinks > > > Key: HDFS-254 > URL: https://issues.apache.org/jira/browse/HDFS-254 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: dhruba borthakur >Assignee: Eli Collins > > HADOOP-4044 introduces symbolic links in HDFS. That patch has some unit tests > associated with it. The aim of this JIRA is to add more unit tests (including > negative tests): > deleting a symlink that points to a file > deleting a symlink that points to a directory > symlink to a symlink > any test cases with relative links > delete linked to file, but not link, access link > delete linked to file, recreate the same file, access link > links to other filesystems > archives containing links > rename a file to an existing symlink > rename a symlink to an existing file > setTimes and setReplication cases should check the result on both the symlink > and linked to file > tests involving lack of permissions to create symlinks > tests that try to createSymlinks with filesystems that don't support them > ... -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-265) Revisit append
[ https://issues.apache.org/jira/browse/HDFS-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755181#action_12755181 ] ryan rawson commented on HDFS-265: -- hey guys, it's been 4 months since this issue started moving, can we get some source so we can review and possibly test? Thanks! > Revisit append > -- > > Key: HDFS-265 > URL: https://issues.apache.org/jira/browse/HDFS-265 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: appendDesign.pdf, appendDesign.pdf, appendDesign1.pdf, > appendDesign2.pdf, AppendSpec.pdf, AppendTestPlan.html, AppendTestPlan.html, > AppendTestPlan.html, AppendTestPlan.html, AppendTestPlan.html, > AppendTestPlan.html, AppendTestPlan.html, TestPlanAppend.html > > > HADOOP-1700 and related issues have put a lot of efforts to provide the first > implementation of append. However, append is such a complex feature. It turns > out that there are issues that were initially seemed trivial but needs a > careful design. This jira revisits append, aiming for a design and > implementation supporting a semantics that are acceptable to its users. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-589) Change block write protocol to support pipeline recovery
[ https://issues.apache.org/jira/browse/HDFS-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-589: --- Attachment: opWriteProtocol1.patch The attached patch addressed all review comments. It also addressed a findbug warning: REC Exception is caught when Exception is not thrown in org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer$ResponseProcessor.run() Since the responder thread is designed to handle RuntimeException, I put this warning in the excluded findbug warning list. > Change block write protocol to support pipeline recovery > > > Key: HDFS-589 > URL: https://issues.apache.org/jira/browse/HDFS-589 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: opWriteProtocol.patch, opWriteProtocol1.patch > > > Current block write operation's header has the following fields: > blockId blockGS pipelineSize isRecovery clientName hasSource source > #datanodesInDownStreamPipeline downstreamDatanodes > I'd like to change the header to be > blockId blockGS pipelineSize clientName flags blockMinLen blockMaxLen newGS > hasSource source #datanodesInDownStreamPipeline downstreamDatanodes > With this protocol change, pipeline recovery will be performed when a mew > pipeline is set up. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-265) Revisit append
[ https://issues.apache.org/jira/browse/HDFS-265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755185#action_12755185 ] Konstantin Boudnik commented on HDFS-265: - Certainly, it could be found in the append branch at https://svn.apache.org/repos/asf/hadoop/hdfs/branches/HDFS-265 > Revisit append > -- > > Key: HDFS-265 > URL: https://issues.apache.org/jira/browse/HDFS-265 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: appendDesign.pdf, appendDesign.pdf, appendDesign1.pdf, > appendDesign2.pdf, AppendSpec.pdf, AppendTestPlan.html, AppendTestPlan.html, > AppendTestPlan.html, AppendTestPlan.html, AppendTestPlan.html, > AppendTestPlan.html, AppendTestPlan.html, TestPlanAppend.html > > > HADOOP-1700 and related issues have put a lot of efforts to provide the first > implementation of append. However, append is such a complex feature. It turns > out that there are issues that were initially seemed trivial but needs a > careful design. This jira revisits append, aiming for a design and > implementation supporting a semantics that are acceptable to its users. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-589) Change block write protocol to support pipeline recovery
[ https://issues.apache.org/jira/browse/HDFS-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755184#action_12755184 ] Hairong Kuang commented on HDFS-589: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. > Change block write protocol to support pipeline recovery > > > Key: HDFS-589 > URL: https://issues.apache.org/jira/browse/HDFS-589 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: opWriteProtocol.patch, opWriteProtocol1.patch > > > Current block write operation's header has the following fields: > blockId blockGS pipelineSize isRecovery clientName hasSource source > #datanodesInDownStreamPipeline downstreamDatanodes > I'd like to change the header to be > blockId blockGS pipelineSize clientName flags blockMinLen blockMaxLen newGS > hasSource source #datanodesInDownStreamPipeline downstreamDatanodes > With this protocol change, pipeline recovery will be performed when a mew > pipeline is set up. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-592) Allow client to get a new generation stamp from NameNode
[ https://issues.apache.org/jira/browse/HDFS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755193#action_12755193 ] Kan Zhang commented on HDFS-592: As I explained before, method name like getNewStampAndToken() is misleading in the sense that it looks like it's a general method that any client can call to get a token. Actually, it's not. Only a client who is doing pipeline recovery (or already holding the lease) is supposed to use that method. It doesn't have to be the only thing that the client does when doing a recovery. Having said that, I'm open to other names that properly capture the context. > Allow client to get a new generation stamp from NameNode > > > Key: HDFS-592 > URL: https://issues.apache.org/jira/browse/HDFS-592 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: newGS.patch, newGS1.patch > > > This issue aims to add an API to ClientProtocol that fetches a new > generation stamp and an access token from NameNode to support append or > pipeline recovery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755196#action_12755196 ] Hadoop QA commented on HDFS-617: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419552/h617-01.patch against trunk revision 814449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/console This message is automatically generated. > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-606) ConcurrentModificationException in invalidateCorruptReplicas()
[
https://issues.apache.org/jira/browse/HDFS-606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755201#action_12755201
]
Hudson commented on HDFS-606:
-
Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #26 (See
[http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/])
> ConcurrentModificationException in invalidateCorruptReplicas()
> --
>
> Key: HDFS-606
> URL: https://issues.apache.org/jira/browse/HDFS-606
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: CMEinCorruptReplicas.patch
>
>
> {{BlockManager.invalidateCorruptReplicas()}} iterates over
> DatanodeDescriptor-s while removing corrupt replicas from the descriptors.
> This causes {{ConcurrentModificationException}} if there is more than one
> replicas of the block. I ran into this exception debugging different
> scenarios in append, but it should be fixed in the trunk too.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-472) Document hdfsproxy design and set-up guide
[ https://issues.apache.org/jira/browse/HDFS-472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755199#action_12755199 ] Hudson commented on HDFS-472: - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #26 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/]) > Document hdfsproxy design and set-up guide > -- > > Key: HDFS-472 > URL: https://issues.apache.org/jira/browse/HDFS-472 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/hdfsproxy >Reporter: zhiyong zhang >Assignee: zhiyong zhang > Fix For: 0.21.0 > > Attachments: HDFS-472.patch, HDFS-472.patch, HDFS-472.patch, > hdfsproxy.pdf, hdfsproxy.pdf > > > currently hdfsproxy only have a README file that does not follow closely to > the code. Need more documentation on the design, build and set-up guide. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-602) Atempt to make a directory under an existing file on DistributedFileSystem should throw an FileAlreadyExistsException instead of FileNotFoundException
[ https://issues.apache.org/jira/browse/HDFS-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755202#action_12755202 ] Hudson commented on HDFS-602: - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #26 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/]) > Atempt to make a directory under an existing file on DistributedFileSystem > should throw an FileAlreadyExistsException instead of FileNotFoundException > -- > > Key: HDFS-602 > URL: https://issues.apache.org/jira/browse/HDFS-602 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client, name-node >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Fix For: 0.21.0 > > Attachments: HDFS-602.patch > > > Atempt to make a directory under an existing file on DistributedFileSystem > should throw an FileAlreadyExistsException instead of FileNotFoundException. > Also we should unwrap this exception from RemoteException -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-614) TestDatanodeBlockScanner obtain should data-node directories directly from MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755204#action_12755204
]
Hudson commented on HDFS-614:
-
Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #26 (See
[http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/])
> TestDatanodeBlockScanner obtain should data-node directories directly from
> MiniDFSCluster
> -
>
> Key: HDFS-614
> URL: https://issues.apache.org/jira/browse/HDFS-614
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 0.21.0
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: TestDNBlockScanner.patch
>
>
> TestDatanodeBlockScanner relies on that data-node directories are listed in
> {{test.build.data}}, which is not true if the test run from eclipse. It shold
> get the directories directly from {{MiniDFSCluster}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-612) FSDataset should not use org.mortbay.log.Log
[ https://issues.apache.org/jira/browse/HDFS-612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755203#action_12755203 ] Hudson commented on HDFS-612: - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #26 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/]) > FSDataset should not use org.mortbay.log.Log > > > Key: HDFS-612 > URL: https://issues.apache.org/jira/browse/HDFS-612 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.21.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.21.0 > > Attachments: h612_20090911.patch, h612_20090911b.patch > > > There are some codes in FSDataset using org.mortbay.log.Log. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-601) TestBlockReport should obtain data directories from MiniHDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755205#action_12755205 ] Hudson commented on HDFS-601: - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #26 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/26/]) > TestBlockReport should obtain data directories from MiniHDFSCluster > --- > > Key: HDFS-601 > URL: https://issues.apache.org/jira/browse/HDFS-601 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.21.0, Append Branch >Reporter: Konstantin Shvachko >Assignee: Konstantin Boudnik > Fix For: 0.21.0 > > Attachments: HDFS-601.patch, HDFS-601.patch > > > TestBlockReport relies on that "test.build.data" property is set in > configuration, which is not always correct, e.g. when you run test from > eclipse. It would be better to get data directories directly from the > mini-cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-592) Allow client to get a new generation stamp from NameNode
[ https://issues.apache.org/jira/browse/HDFS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755206#action_12755206 ] Kan Zhang commented on HDFS-592: > Only a client who is doing pipeline recovery (or already holding the lease) Actually, "already holding the lease" is not a good description either since a client who is writing a new block (and hence holding the lease) shouldn't use this method to get token. > Allow client to get a new generation stamp from NameNode > > > Key: HDFS-592 > URL: https://issues.apache.org/jira/browse/HDFS-592 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: newGS.patch, newGS1.patch > > > This issue aims to add an API to ClientProtocol that fetches a new > generation stamp and an access token from NameNode to support append or > pipeline recovery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-589) Change block write protocol to support pipeline recovery
[ https://issues.apache.org/jira/browse/HDFS-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-589: Component/s: hdfs client data-node Hadoop Flags: [Reviewed] +1 patch looks good > Change block write protocol to support pipeline recovery > > > Key: HDFS-589 > URL: https://issues.apache.org/jira/browse/HDFS-589 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, hdfs client >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: opWriteProtocol.patch, opWriteProtocol1.patch > > > Current block write operation's header has the following fields: > blockId blockGS pipelineSize isRecovery clientName hasSource source > #datanodesInDownStreamPipeline downstreamDatanodes > I'd like to change the header to be > blockId blockGS pipelineSize clientName flags blockMinLen blockMaxLen newGS > hasSource source #datanodesInDownStreamPipeline downstreamDatanodes > With this protocol change, pipeline recovery will be performed when a mew > pipeline is set up. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-618) Support for non-recursive mkdir in HDFS
[ https://issues.apache.org/jira/browse/HDFS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-618: --- Attachment: h618-03.patch > Support for non-recursive mkdir in HDFS > --- > > Key: HDFS-618 > URL: https://issues.apache.org/jira/browse/HDFS-618 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h618-03.patch > > > Existing mkdirs call automatically creates missing parent directories. > HADOOP-4952 call for a mkdir call that doesn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-618) Support for non-recursive mkdir in HDFS
[ https://issues.apache.org/jira/browse/HDFS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755231#action_12755231 ] Kan Zhang commented on HDFS-618: Attached a patch that 1. Added an interface method mkdir to DistributedFileSystem. It throws FileNotFoundException when parent dir doesn't exist. 2. Added a boolean param createParent to mkdirs() RPC call on ClientProtocol. > Support for non-recursive mkdir in HDFS > --- > > Key: HDFS-618 > URL: https://issues.apache.org/jira/browse/HDFS-618 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h618-03.patch > > > Existing mkdirs call automatically creates missing parent directories. > HADOOP-4952 call for a mkdir call that doesn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-618) Support for non-recursive mkdir in HDFS
[ https://issues.apache.org/jira/browse/HDFS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-618: --- Status: Patch Available (was: Open) > Support for non-recursive mkdir in HDFS > --- > > Key: HDFS-618 > URL: https://issues.apache.org/jira/browse/HDFS-618 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h618-03.patch > > > Existing mkdirs call automatically creates missing parent directories. > HADOOP-4952 call for a mkdir call that doesn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-619) Support replica recovery initialization in datanode
Support replica recovery initialization in datanode --- Key: HDFS-619 URL: https://issues.apache.org/jira/browse/HDFS-619 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Affects Versions: Append Branch Reporter: Tsz Wo (Nicholas), SZE Fix For: Append Branch In the [append design|https://issues.apache.org/jira/secure/attachment/12415768/appendDesign2.pdf], there is a block recovery algorithm presented in Section 6.5. We are going to implement step 4a, datanode replica recovery initialization, in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-619) Support replica recovery initialization in datanode
[ https://issues.apache.org/jira/browse/HDFS-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-619: Attachment: h619_20090914.patch h619_20090914.patch: a preliminary patch for reviewing. - not sure how to correctly stop writers. - need new unit tests. > Support replica recovery initialization in datanode > --- > > Key: HDFS-619 > URL: https://issues.apache.org/jira/browse/HDFS-619 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: Append Branch >Reporter: Tsz Wo (Nicholas), SZE > Fix For: Append Branch > > Attachments: h619_20090914.patch > > > In the [append > design|https://issues.apache.org/jira/secure/attachment/12415768/appendDesign2.pdf], > there is a block recovery algorithm presented in Section 6.5. We are going > to implement step 4a, datanode replica recovery initialization, in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-617) Support for non-recursive create() in HDFS
[
https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755240#action_12755240
]
Tsz Wo (Nicholas), SZE commented on HDFS-617:
-
Patch looks good except that createParent should be false when for append.
{code}
startFileInternal(src, null, holder, clientMachine,
EnumSet.of(CreateFlag.APPEND),
- (short)blockManager.maxReplication, (long)0);
+ true, (short)blockManager.maxReplication, (long)0);
getEditLog().logSync();
{code}
> Support for non-recursive create() in HDFS
> --
>
> Key: HDFS-617
> URL: https://issues.apache.org/jira/browse/HDFS-617
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kan Zhang
>Assignee: Kan Zhang
> Attachments: h617-01.patch
>
>
> HADOOP-4952 calls for a create call that doesn't automatically create missing
> parent directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-589) Change block write protocol to support pipeline recovery
[ https://issues.apache.org/jira/browse/HDFS-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755242#action_12755242 ] Hairong Kuang commented on HDFS-589: Ant test-core passed except for TestBackupNode. > Change block write protocol to support pipeline recovery > > > Key: HDFS-589 > URL: https://issues.apache.org/jira/browse/HDFS-589 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, hdfs client >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: opWriteProtocol.patch, opWriteProtocol1.patch > > > Current block write operation's header has the following fields: > blockId blockGS pipelineSize isRecovery clientName hasSource source > #datanodesInDownStreamPipeline downstreamDatanodes > I'd like to change the header to be > blockId blockGS pipelineSize clientName flags blockMinLen blockMaxLen newGS > hasSource source #datanodesInDownStreamPipeline downstreamDatanodes > With this protocol change, pipeline recovery will be performed when a mew > pipeline is set up. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Attachment: h617-02.patch > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-618) Support for non-recursive mkdir in HDFS
[
https://issues.apache.org/jira/browse/HDFS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755244#action_12755244
]
Tsz Wo (Nicholas), SZE commented on HDFS-618:
-
- new public methods needs javadoc, e.g.
{code}
+ public boolean mkdir(Path f, FsPermission permission) throws IOException {
+return dfs.mkdirs(getPathName(f), permission, false);
+ }
{code}
- The following method in DFSClient is not used. How about we deprecate it?
{code}
public boolean mkdirs(String src) throws IOException {
-return mkdirs(src, null);
+return mkdirs(src, null, true);
}
{code}
- This and HDFS-617 are going to conflict each other. It looks like that this
patch is simpler and may go first. BTW, could you create helper methods for
the common codes like the following?
{code}
+if (!createParent) {
+ Path parent = new Path(src).getParent();
+ if (parent != null && !dir.isDir(parent.toString())) {
+throw new FileNotFoundException("Can't mkdir " + src
++ "; parent directory " + parent.toString() + " doesn't exist.");
+ }
+}
{code}
> Support for non-recursive mkdir in HDFS
> ---
>
> Key: HDFS-618
> URL: https://issues.apache.org/jira/browse/HDFS-618
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kan Zhang
>Assignee: Kan Zhang
> Attachments: h618-03.patch
>
>
> Existing mkdirs call automatically creates missing parent directories.
> HADOOP-4952 call for a mkdir call that doesn't.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Status: Patch Available (was: Open) > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Status: Open (was: Patch Available) > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755246#action_12755246 ] Kan Zhang commented on HDFS-617: Addressed Nicholas' comment with new patch. > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-589) Change block write protocol to support pipeline recovery
[ https://issues.apache.org/jira/browse/HDFS-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang resolved HDFS-589. Resolution: Fixed I've just committed this. > Change block write protocol to support pipeline recovery > > > Key: HDFS-589 > URL: https://issues.apache.org/jira/browse/HDFS-589 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, hdfs client >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: opWriteProtocol.patch, opWriteProtocol1.patch > > > Current block write operation's header has the following fields: > blockId blockGS pipelineSize isRecovery clientName hasSource source > #datanodesInDownStreamPipeline downstreamDatanodes > I'd like to change the header to be > blockId blockGS pipelineSize clientName flags blockMinLen blockMaxLen newGS > hasSource source #datanodesInDownStreamPipeline downstreamDatanodes > With this protocol change, pipeline recovery will be performed when a mew > pipeline is set up. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-592) Allow client to get a new generation stamp from NameNode
[ https://issues.apache.org/jira/browse/HDFS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755252#action_12755252 ] Hairong Kuang commented on HDFS-592: As I said, the method is not used only for pipeline recovery, it is also used for setting up the initial pipeline for append as well. So pipelineRecovery is not an option. It's hard for a method name to reflect the context under which a method is called. I don't think any of the client method ever does this. How about I putting the context in the java doc? > Allow client to get a new generation stamp from NameNode > > > Key: HDFS-592 > URL: https://issues.apache.org/jira/browse/HDFS-592 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: newGS.patch, newGS1.patch > > > This issue aims to add an API to ClientProtocol that fetches a new > generation stamp and an access token from NameNode to support append or > pipeline recovery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-592) Allow client to get a new generation stamp from NameNode
[ https://issues.apache.org/jira/browse/HDFS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755258#action_12755258 ] Kan Zhang commented on HDFS-592: > I don't think any of the client method ever does this. I think create(), append(), addBlock() all capture their context pretty well. Although pipelineRecovery() is not perfect, it's still much better than getNewStampAndToken() which doesn't tell any context at all. > Allow client to get a new generation stamp from NameNode > > > Key: HDFS-592 > URL: https://issues.apache.org/jira/browse/HDFS-592 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: newGS.patch, newGS1.patch > > > This issue aims to add an API to ClientProtocol that fetches a new > generation stamp and an access token from NameNode to support append or > pipeline recovery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Attachment: h617-03.patch > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch, h617-03.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755262#action_12755262 ] Kan Zhang commented on HDFS-617: attached a new patch that factors out the code for verifying parent dir exists. > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch, h617-03.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-610) Add support for FileContext
[ https://issues.apache.org/jira/browse/HDFS-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-610: -- Attachment: FileContext-common14.patch > Add support for FileContext > --- > > Key: HDFS-610 > URL: https://issues.apache.org/jira/browse/HDFS-610 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs client, name-node >Affects Versions: 0.21.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: FileContext-hdfs12.patch > > > Add support for FileContext (linked jira) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-610) Add support for FileContext
[ https://issues.apache.org/jira/browse/HDFS-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-610: -- Attachment: (was: FileContext-common14.patch) > Add support for FileContext > --- > > Key: HDFS-610 > URL: https://issues.apache.org/jira/browse/HDFS-610 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs client, name-node >Affects Versions: 0.21.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: FileContext-hdfs12.patch > > > Add support for FileContext (linked jira) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-610) Add support for FileContext
[ https://issues.apache.org/jira/browse/HDFS-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-610: -- Attachment: FileContext-hdfs14.patch > Add support for FileContext > --- > > Key: HDFS-610 > URL: https://issues.apache.org/jira/browse/HDFS-610 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs client, name-node >Affects Versions: 0.21.0 >Reporter: Sanjay Radia >Assignee: Sanjay Radia > Attachments: FileContext-hdfs12.patch, FileContext-hdfs14.patch > > > Add support for FileContext (linked jira) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-618) Support for non-recursive mkdir in HDFS
[ https://issues.apache.org/jira/browse/HDFS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-618: --- Attachment: h618-04.patch > Support for non-recursive mkdir in HDFS > --- > > Key: HDFS-618 > URL: https://issues.apache.org/jira/browse/HDFS-618 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h618-03.patch, h618-04.patch > > > Existing mkdirs call automatically creates missing parent directories. > HADOOP-4952 call for a mkdir call that doesn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-618) Support for non-recursive mkdir in HDFS
[ https://issues.apache.org/jira/browse/HDFS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-618: --- Status: Open (was: Patch Available) > Support for non-recursive mkdir in HDFS > --- > > Key: HDFS-618 > URL: https://issues.apache.org/jira/browse/HDFS-618 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h618-03.patch, h618-04.patch > > > Existing mkdirs call automatically creates missing parent directories. > HADOOP-4952 call for a mkdir call that doesn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-618) Support for non-recursive mkdir in HDFS
[ https://issues.apache.org/jira/browse/HDFS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-618: --- Status: Patch Available (was: Open) > Support for non-recursive mkdir in HDFS > --- > > Key: HDFS-618 > URL: https://issues.apache.org/jira/browse/HDFS-618 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h618-03.patch, h618-04.patch > > > Existing mkdirs call automatically creates missing parent directories. > HADOOP-4952 call for a mkdir call that doesn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Status: Open (was: Patch Available) > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch, h617-03.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Status: Patch Available (was: Open) > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch, h617-03.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-618) Support for non-recursive mkdir in HDFS
[ https://issues.apache.org/jira/browse/HDFS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755266#action_12755266 ] Kan Zhang commented on HDFS-618: uploaded a new patch that adds the javadoc and deprecated mkdirs(String). Will upload a new patch after HDFS-617 is in, to resolve any conflict and also re-use the factored out verifyParentDir code. > Support for non-recursive mkdir in HDFS > --- > > Key: HDFS-618 > URL: https://issues.apache.org/jira/browse/HDFS-618 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h618-03.patch, h618-04.patch > > > Existing mkdirs call automatically creates missing parent directories. > HADOOP-4952 call for a mkdir call that doesn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Attachment: h617-04.patch one minor javadoc change. > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch, h617-03.patch, > h617-04.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Status: Patch Available (was: Open) > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch, h617-03.patch, > h617-04.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kan Zhang updated HDFS-617: --- Status: Open (was: Patch Available) > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch, h617-03.patch, > h617-04.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-592) Allow client to get a new generation stamp from NameNode
[ https://issues.apache.org/jira/browse/HDFS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755273#action_12755273 ] Hairong Kuang commented on HDFS-592: The problem is that getting a new stamp and token is not pipeline recovery. The result is used for recovering a pipeline. So it is not an option. I am open if you could come up with other options. > Allow client to get a new generation stamp from NameNode > > > Key: HDFS-592 > URL: https://issues.apache.org/jira/browse/HDFS-592 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: newGS.patch, newGS1.patch > > > This issue aims to add an API to ClientProtocol that fetches a new > generation stamp and an access token from NameNode to support append or > pipeline recovery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-618) Support for non-recursive mkdir in HDFS
[ https://issues.apache.org/jira/browse/HDFS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755277#action_12755277 ] Hadoop QA commented on HDFS-618: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419577/h618-03.patch against trunk revision 814449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/27/console This message is automatically generated. > Support for non-recursive mkdir in HDFS > --- > > Key: HDFS-618 > URL: https://issues.apache.org/jira/browse/HDFS-618 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h618-03.patch, h618-04.patch > > > Existing mkdirs call automatically creates missing parent directories. > HADOOP-4952 call for a mkdir call that doesn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-592) Allow client to get a new generation stamp from NameNode
[ https://issues.apache.org/jira/browse/HDFS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755280#action_12755280 ] Kan Zhang commented on HDFS-592: > The problem is that getting a new stamp and token is not pipeline recovery. > The result is used for recovering a pipeline. You can say the same about addBlock(), can't you? > Allow client to get a new generation stamp from NameNode > > > Key: HDFS-592 > URL: https://issues.apache.org/jira/browse/HDFS-592 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: newGS.patch, newGS1.patch > > > This issue aims to add an API to ClientProtocol that fetches a new > generation stamp and an access token from NameNode to support append or > pipeline recovery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-619) Support replica recovery initialization in datanode
[ https://issues.apache.org/jira/browse/HDFS-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-619: Attachment: h619_20090914b.patch h619_20090914b.patch: added some tests and changed the codes a little bit. - will add more tests. - There are quite a few constructors in ReplicaInfo, ReplicaBeingWritten, etc. Should we simplify them? > Support replica recovery initialization in datanode > --- > > Key: HDFS-619 > URL: https://issues.apache.org/jira/browse/HDFS-619 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: Append Branch >Reporter: Tsz Wo (Nicholas), SZE > Fix For: Append Branch > > Attachments: h619_20090914.patch, h619_20090914b.patch > > > In the [append > design|https://issues.apache.org/jira/secure/attachment/12415768/appendDesign2.pdf], > there is a block recovery algorithm presented in Section 6.5. We are going > to implement step 4a, datanode replica recovery initialization, in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-592) Allow client to get a new generation stamp from NameNode
[ https://issues.apache.org/jira/browse/HDFS-592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755288#action_12755288 ] Kan Zhang commented on HDFS-592: In the spirit of addBlock(), how about we use recoverPipeline()? I guess what happens here is you don't want add a new appendBlock() call for the initial pipeline setup for append and instead want to re-use the recoverPipeline() call, which is fine. However, in the absence of a better name for both recoverPipeline and appendBlock, I can live with recoverPipeline than something totally non-descriptive, like getNewStampAndToken(). > Allow client to get a new generation stamp from NameNode > > > Key: HDFS-592 > URL: https://issues.apache.org/jira/browse/HDFS-592 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: newGS.patch, newGS1.patch > > > This issue aims to add an API to ClientProtocol that fetches a new > generation stamp and an access token from NameNode to support append or > pipeline recovery. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755302#action_12755302 ] Hadoop QA commented on HDFS-617: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419582/h617-02.patch against trunk revision 814449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/console This message is automatically generated. > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch, h617-03.patch, > h617-04.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-605) There's not need to run fault-inject tests by 'run-test-hdfs-with-mr' target
[
https://issues.apache.org/jira/browse/HDFS-605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755308#action_12755308
]
Hudson commented on HDFS-605:
-
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See
[http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/])
. Do not run fault injection tests in the run-test-hdfs-with-mr target.
Contributed by Konstantin Boudnik
> There's not need to run fault-inject tests by 'run-test-hdfs-with-mr' target
>
>
> Key: HDFS-605
> URL: https://issues.apache.org/jira/browse/HDFS-605
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: build, test
>Affects Versions: 0.21.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Fix For: 0.21.0
>
> Attachments: HDFS-605.patch
>
>
> It turns out that running fault injection tests doesn't make any sense when
> {{run-test-hdfs-with-mr}} target is being executed. Thus, {{build.xml}} has
> to be modified.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-472) Document hdfsproxy design and set-up guide
[ https://issues.apache.org/jira/browse/HDFS-472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755305#action_12755305 ] Hudson commented on HDFS-472: - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/]) . Update hdfsproxy documentation. Adds a setup guide and design document. Contributed by Zhiyong Zhang > Document hdfsproxy design and set-up guide > -- > > Key: HDFS-472 > URL: https://issues.apache.org/jira/browse/HDFS-472 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/hdfsproxy >Reporter: zhiyong zhang >Assignee: zhiyong zhang > Fix For: 0.21.0 > > Attachments: HDFS-472.patch, HDFS-472.patch, HDFS-472.patch, > hdfsproxy.pdf, hdfsproxy.pdf > > > currently hdfsproxy only have a README file that does not follow closely to > the code. Need more documentation on the design, build and set-up guide. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-614) TestDatanodeBlockScanner obtain should data-node directories directly from MiniDFSCluster
[
https://issues.apache.org/jira/browse/HDFS-614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755312#action_12755312
]
Hudson commented on HDFS-614:
-
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See
[http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/])
. TestDatanodeBlockScanner obtains data directories directly from
MiniHDFSCluster. Contributed by Konstantin Shvachko.
> TestDatanodeBlockScanner obtain should data-node directories directly from
> MiniDFSCluster
> -
>
> Key: HDFS-614
> URL: https://issues.apache.org/jira/browse/HDFS-614
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 0.21.0
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: TestDNBlockScanner.patch
>
>
> TestDatanodeBlockScanner relies on that data-node directories are listed in
> {{test.build.data}}, which is not true if the test run from eclipse. It shold
> get the directories directly from {{MiniDFSCluster}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-601) TestBlockReport should obtain data directories from MiniHDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755311#action_12755311 ] Hudson commented on HDFS-601: - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/]) . TestBlockReport obtains data directories directly from MiniHDFSCluster. Contributed by Konstantin Boudnik. > TestBlockReport should obtain data directories from MiniHDFSCluster > --- > > Key: HDFS-601 > URL: https://issues.apache.org/jira/browse/HDFS-601 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.21.0, Append Branch >Reporter: Konstantin Shvachko >Assignee: Konstantin Boudnik > Fix For: 0.21.0 > > Attachments: HDFS-601.patch, HDFS-601.patch > > > TestBlockReport relies on that "test.build.data" property is set in > configuration, which is not always correct, e.g. when you run test from > eclipse. It would be better to get data directories directly from the > mini-cluster. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-606) ConcurrentModificationException in invalidateCorruptReplicas()
[
https://issues.apache.org/jira/browse/HDFS-606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755307#action_12755307
]
Hudson commented on HDFS-606:
-
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See
[http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/])
. Fix ConcurrentModificationException in invalidateCorruptReplicas().
Contributed by Konstantin Shvachko.
> ConcurrentModificationException in invalidateCorruptReplicas()
> --
>
> Key: HDFS-606
> URL: https://issues.apache.org/jira/browse/HDFS-606
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: CMEinCorruptReplicas.patch
>
>
> {{BlockManager.invalidateCorruptReplicas()}} iterates over
> DatanodeDescriptor-s while removing corrupt replicas from the descriptors.
> This causes {{ConcurrentModificationException}} if there is more than one
> replicas of the block. I ran into this exception debugging different
> scenarios in append, but it should be fixed in the trunk too.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-612) FSDataset should not use org.mortbay.log.Log
[ https://issues.apache.org/jira/browse/HDFS-612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755310#action_12755310 ] Hudson commented on HDFS-612: - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/]) . Remove the use of org.mortbay.log.Log in FSDataset. > FSDataset should not use org.mortbay.log.Log > > > Key: HDFS-612 > URL: https://issues.apache.org/jira/browse/HDFS-612 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.21.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.21.0 > > Attachments: h612_20090911.patch, h612_20090911b.patch > > > There are some codes in FSDataset using org.mortbay.log.Log. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-602) Atempt to make a directory under an existing file on DistributedFileSystem should throw an FileAlreadyExistsException instead of FileNotFoundException
[ https://issues.apache.org/jira/browse/HDFS-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755309#action_12755309 ] Hudson commented on HDFS-602: - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #6 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/6/]) . DistributedFileSystem mkdirs command throws FileAlreadyExistsException instead of FileNotFoundException. Contributed by Boris Shkolnik. > Atempt to make a directory under an existing file on DistributedFileSystem > should throw an FileAlreadyExistsException instead of FileNotFoundException > -- > > Key: HDFS-602 > URL: https://issues.apache.org/jira/browse/HDFS-602 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client, name-node >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Fix For: 0.21.0 > > Attachments: HDFS-602.patch > > > Atempt to make a directory under an existing file on DistributedFileSystem > should throw an FileAlreadyExistsException instead of FileNotFoundException. > Also we should unwrap this exception from RemoteException -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755326#action_12755326 ] eric baldeschwieler commented on HDFS-245: -- Hi Folks, Expect more discussion here! I'll let others express their opinions, but just FYI, there are unresolved design issues here that folks on our HDFS team feel strongly about... E14 > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, HADOOP-4044-strawman.patch, > symlink-0.20.0.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-618) Support for non-recursive mkdir in HDFS
[ https://issues.apache.org/jira/browse/HDFS-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755337#action_12755337 ] Hadoop QA commented on HDFS-618: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419591/h618-04.patch against trunk revision 814449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/28/console This message is automatically generated. > Support for non-recursive mkdir in HDFS > --- > > Key: HDFS-618 > URL: https://issues.apache.org/jira/browse/HDFS-618 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h618-03.patch, h618-04.patch > > > Existing mkdirs call automatically creates missing parent directories. > HADOOP-4952 call for a mkdir call that doesn't. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-617) Support for non-recursive create() in HDFS
[ https://issues.apache.org/jira/browse/HDFS-617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755353#action_12755353 ] Hadoop QA commented on HDFS-617: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419593/h617-04.patch against trunk revision 814449. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/7/console This message is automatically generated. > Support for non-recursive create() in HDFS > -- > > Key: HDFS-617 > URL: https://issues.apache.org/jira/browse/HDFS-617 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: h617-01.patch, h617-02.patch, h617-03.patch, > h617-04.patch > > > HADOOP-4952 calls for a create call that doesn't automatically create missing > parent directories. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-385: -- Tags: (was: fb) Resolution: Fixed Release Note: An experimental API that allows an module external to HDFS to specify the policy that HDFS should use to allocate new blocks replicas of a file. Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. > Design a pluggable interface to place replicas of blocks in HDFS > > > Key: HDFS-385 > URL: https://issues.apache.org/jira/browse/HDFS-385 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: BlockPlacementPluggable.txt, > BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, > BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, > BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, > BlockPlacementPluggable7.txt > > > The current HDFS code typically places one replica on local rack, the second > replica on remote random rack and the third replica on a random node of that > remote rack. This algorithm is baked in the NameNode's code. It would be nice > to make the block placement algorithm a pluggable interface. This will allow > experimentation of different placement algorithms based on workloads, > availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-472) Document hdfsproxy design and set-up guide
[ https://issues.apache.org/jira/browse/HDFS-472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755387#action_12755387 ] Hudson commented on HDFS-472: - Integrated in Hadoop-Hdfs-trunk-Commit #34 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/34/]) > Document hdfsproxy design and set-up guide > -- > > Key: HDFS-472 > URL: https://issues.apache.org/jira/browse/HDFS-472 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/hdfsproxy >Reporter: zhiyong zhang >Assignee: zhiyong zhang > Fix For: 0.21.0 > > Attachments: HDFS-472.patch, HDFS-472.patch, HDFS-472.patch, > hdfsproxy.pdf, hdfsproxy.pdf > > > currently hdfsproxy only have a README file that does not follow closely to > the code. Need more documentation on the design, build and set-up guide. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755388#action_12755388 ] Hudson commented on HDFS-385: - Integrated in Hadoop-Hdfs-trunk-Commit #34 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/34/]) . Add support for an API that allows a module external to HDFS to specify how HDFS blocks should be placed. (dhruba) > Design a pluggable interface to place replicas of blocks in HDFS > > > Key: HDFS-385 > URL: https://issues.apache.org/jira/browse/HDFS-385 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: BlockPlacementPluggable.txt, > BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, > BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, > BlockPlacementPluggable5.txt, BlockPlacementPluggable6.txt, > BlockPlacementPluggable7.txt > > > The current HDFS code typically places one replica on local rack, the second > replica on remote random rack and the third replica on a random node of that > remote rack. This algorithm is baked in the NameNode's code. It would be nice > to make the block placement algorithm a pluggable interface. This will allow > experimentation of different placement algorithms based on workloads, > availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
