[jira] Commented: (HDFS-531) Renaming of configuration keys
[ https://issues.apache.org/jira/browse/HDFS-531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742311#action_12742311 ] Tom White commented on HDFS-531: * It seems wrong that there are HDFS properties in Common. Rather than dfs.networktopology.* how about net.topology.* to reflect the fact they are in the o.a.h.net package? Also, why not keep hadoop.tmp.dir instead of renaming it dfs.tmp.dir? * The distributed cache is moving to MapReduce so filecache.local.cache.size, should have a mapred prefix. See also MAPREDUCE-711. * My preference would be that the HDFS prefix is hdfs, not dfs, to be consistent with the protocol scheme (hdfs://) and package names (o.a.h.hdfs). But this is a big change. > Renaming of configuration keys > -- > > Key: HDFS-531 > URL: https://issues.apache.org/jira/browse/HDFS-531 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.21.0 > > Attachments: changed_config_keys.txt > > > Keys in configuration files should be standardized so that key names reflect > the components they are used in. > For example: >dfs.backup.address should be renamed to dfs.namenode.backup.address >dfs.data.dir should be renamed to dfs.datanode.data.dir > This change will impact both hdfs and common sources. > Following convention is proposed: > 1. Any key related hdfs should begin with 'dfs'. > 2. Any key related to namenode, datanode or client should begin with > dfs.namenode, dfs.datanode or dfs.client respectively. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-528) Add ability for safemode to wait for a minimum number of live datanodes
[ https://issues.apache.org/jira/browse/HDFS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742339#action_12742339 ] Steve Loughran commented on HDFS-528: - I do something like this in my workflow; block until the the #of workers is above the required number. for one thing, that number could be >1, for another, 15, for a third 3000. It is also rought the same action regardless of whether what you are waiting for is the NN or the JT; its the number of workers a particular bit of the cluster has to help. Rather than extend safemode, could you have something that just blocks off the NN until its worker count is above what your specific script wants? After all, it is your script that has needs, not the NN. > Add ability for safemode to wait for a minimum number of live datanodes > --- > > Key: HDFS-528 > URL: https://issues.apache.org/jira/browse/HDFS-528 > Project: Hadoop HDFS > Issue Type: New Feature > Components: scripts >Affects Versions: 0.21.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-528-v2.txt, hdfs-528-v3.txt, hdfs-528.txt > > > When starting up a fresh cluster programatically, users often want to wait > until DFS is "writable" before continuing in a script. "dfsadmin -safemode > wait" doesn't quite work for this on a completely fresh cluster, since when > there are 0 blocks on the system, 100% of them are accounted for before any > DNs have reported. > This JIRA is to add a command which waits until a certain number of DNs have > reported as alive to the NN. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-534) Required avro classes are missing
[
https://issues.apache.org/jira/browse/HDFS-534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742350#action_12742350
]
Tom White commented on HDFS-534:
bq. I guess you mean libraries.properties instead of ivysettings.xml.
Yes, I did.
> Required avro classes are missing
> -
>
> Key: HDFS-534
> URL: https://issues.apache.org/jira/browse/HDFS-534
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.21.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.21.0
>
> Attachments: h534_20090810.patch, h534_20090810b.patch,
> h534_20090811.patch
>
>
> Some tests like TestDFSIO are failing:
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/avro/io/DatumReader
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:837)
> at
> org.apache.hadoop.io.serializer.SerializationFactory.add(SerializationFactory.java:66)
> at
> org.apache.hadoop.io.serializer.SerializationFactory.(SerializationFactory.java:58)
> at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:916)
> at
> org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:852)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:385)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:276)
> at org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:134)
> at org.apache.hadoop.fs.TestDFSIO.testIOs(TestDFSIO.java:114)
> at org.apache.hadoop.fs.TestDFSIO.testIOs(TestDFSIO.java:99)
> Caused by: java.lang.ClassNotFoundException: org.apache.avro.io.DatumReader
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-528) Add ability for safemode to wait for a minimum number of live datanodes
[ https://issues.apache.org/jira/browse/HDFS-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742401#action_12742401 ] Todd Lipcon commented on HDFS-528: -- Steve: see Dhruba's comment above. In the case of an entirely empty DFS it makes sense that the NN should stay in safe mode until at least one DN shows up. This will help with some newbie errors where the DFS appears to be running but in fact nothing works right because there are no DNs. It also helps with the issue Dhruba mentioned involving unneeded replication during startup, and is more principled than just picking a large number for dfs.safemode.extension. I've got the bug I mentioned last night fixed and will upload a new patch after I do a bit of "burn in" in our sandbox. > Add ability for safemode to wait for a minimum number of live datanodes > --- > > Key: HDFS-528 > URL: https://issues.apache.org/jira/browse/HDFS-528 > Project: Hadoop HDFS > Issue Type: New Feature > Components: scripts >Affects Versions: 0.21.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-528-v2.txt, hdfs-528-v3.txt, hdfs-528.txt > > > When starting up a fresh cluster programatically, users often want to wait > until DFS is "writable" before continuing in a script. "dfsadmin -safemode > wait" doesn't quite work for this on a completely fresh cluster, since when > there are 0 blocks on the system, 100% of them are accounted for before any > DNs have reported. > This JIRA is to add a command which waits until a certain number of DNs have > reported as alive to the NN. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-525) ListPathsServlet.java uses static SimpleDateFormat that has threading issues
[ https://issues.apache.org/jira/browse/HDFS-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742404#action_12742404 ] Hudson commented on HDFS-525: - Integrated in Hadoop-Hdfs-trunk #47 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/47/]) > ListPathsServlet.java uses static SimpleDateFormat that has threading issues > > > Key: HDFS-525 > URL: https://issues.apache.org/jira/browse/HDFS-525 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.20.1, 0.21.0 > > Attachments: HDFS-525.1.patch, HDFS-525.2.patch, HDFS-525.patch, > HDFS-525.patch, HDFS-525.rel20.patch > > > SimpleDateFormat is not thread safe. Multiple threads accessing the servlet > can cause threading issues -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-451) Test DataTransferProtocol with fault injection
[ https://issues.apache.org/jira/browse/HDFS-451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742405#action_12742405 ] Hudson commented on HDFS-451: - Integrated in Hadoop-Hdfs-trunk #47 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/47/]) > Test DataTransferProtocol with fault injection > -- > > Key: HDFS-451 > URL: https://issues.apache.org/jira/browse/HDFS-451 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE >Priority: Minor > Fix For: 0.21.0 > > Attachments: h451_20090806.patch, h451_20090807.patch > > > DataTransferProtocol is a complicated protocol. It deserves to have some > fault injection tests. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-534) Required avro classes are missing
[
https://issues.apache.org/jira/browse/HDFS-534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742403#action_12742403
]
Hudson commented on HDFS-534:
-
Integrated in Hadoop-Hdfs-trunk #47 (See
[http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/47/])
. Include avro in ivy.
> Required avro classes are missing
> -
>
> Key: HDFS-534
> URL: https://issues.apache.org/jira/browse/HDFS-534
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.21.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.21.0
>
> Attachments: h534_20090810.patch, h534_20090810b.patch,
> h534_20090811.patch
>
>
> Some tests like TestDFSIO are failing:
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/avro/io/DatumReader
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:837)
> at
> org.apache.hadoop.io.serializer.SerializationFactory.add(SerializationFactory.java:66)
> at
> org.apache.hadoop.io.serializer.SerializationFactory.(SerializationFactory.java:58)
> at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:916)
> at
> org.apache.hadoop.io.SequenceFile$Writer.(SequenceFile.java:852)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:385)
> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:276)
> at org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:134)
> at org.apache.hadoop.fs.TestDFSIO.testIOs(TestDFSIO.java:114)
> at org.apache.hadoop.fs.TestDFSIO.testIOs(TestDFSIO.java:99)
> Caused by: java.lang.ClassNotFoundException: org.apache.avro.io.DatumReader
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-532) Allow applications to know that a read request failed because block is missing
[ https://issues.apache.org/jira/browse/HDFS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-532: -- Status: Patch Available (was: Open) > Allow applications to know that a read request failed because block is missing > -- > > Key: HDFS-532 > URL: https://issues.apache.org/jira/browse/HDFS-532 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: BlockMissingException.patch > > > I have an application that has intelligence to retrieve data from alternate > locations if HDFS cannot provide this data. This can happen when data in HDFS > is corrupted or the block is missing. HDFS already throws ChecksumException > if the block is corrupted and throws a generic IOException if the block is > missing. I would like HDFS to throw BlockMissingException when a read request > encounters a block that has no locations associated with it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-540) TestNameNodeMetrics fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nigel Daley updated HDFS-540: - Tags: ygridqa > TestNameNodeMetrics fails intermittently > > > Key: HDFS-540 > URL: https://issues.apache.org/jira/browse/HDFS-540 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.21.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.21.0 > > > TestNameNodeMetrics has strict timing constraint that relies on block > management functionality and can fail intermittently. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-509) Redesisn DataNode volumeMap to include all types of Replicas
[ https://issues.apache.org/jira/browse/HDFS-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-509: --- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've committed this. > Redesisn DataNode volumeMap to include all types of Replicas > > > Key: HDFS-509 > URL: https://issues.apache.org/jira/browse/HDFS-509 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: replicas.patch, replicas1.patch, replicas2.patch, > replicas3.patch > > > With the new append design, replicas at a DataNode could be in one of the > five states: Finalized, ReplicaBeingWritten, ReplicaWaitingToBeRecovered, > ReplicaUnderRecovery, and Temporay. This jira is to > 1. define the in-memory representation of these five types of replicas > 2. redefine volumeMap to include all five types of replicas and remove > ongoingCreates map. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-509) Redesign DataNode volumeMap to include all types of Replicas
[ https://issues.apache.org/jira/browse/HDFS-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-509: --- Summary: Redesign DataNode volumeMap to include all types of Replicas (was: Redesisn DataNode volumeMap to include all types of Replicas) > Redesign DataNode volumeMap to include all types of Replicas > > > Key: HDFS-509 > URL: https://issues.apache.org/jira/browse/HDFS-509 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: replicas.patch, replicas1.patch, replicas2.patch, > replicas3.patch > > > With the new append design, replicas at a DataNode could be in one of the > five states: Finalized, ReplicaBeingWritten, ReplicaWaitingToBeRecovered, > ReplicaUnderRecovery, and Temporay. This jira is to > 1. define the in-memory representation of these five types of replicas > 2. redefine volumeMap to include all five types of replicas and remove > ongoingCreates map. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-409) Add more access token tests
[ https://issues.apache.org/jira/browse/HDFS-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742453#action_12742453 ] Kan Zhang commented on HDFS-409: > What are the changes about? Is there any bug fix? No bug fix and no change of semantics to DFSClient and DataXceiver. 2 package private methods are added to DFSClient to read access token for testing purposes. The rest of the changes are simply changes to logging messages. > Add more access token tests > --- > > Key: HDFS-409 > URL: https://issues.apache.org/jira/browse/HDFS-409 > Project: Hadoop HDFS > Issue Type: Test > Components: data-node, hdfs client >Reporter: Kan Zhang >Assignee: Kan Zhang > Attachments: AccessTokenTestPlan.pdf, h409-23.patch, h409-25.patch, > h409-27.patch, h409-29.patch > > > Need to add more unit tests for access token -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-409) Add more access token tests
[ https://issues.apache.org/jira/browse/HDFS-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742471#action_12742471 ] Tsz Wo (Nicholas), SZE commented on HDFS-409: - > No bug fix and no change of semantics ... That's great. I was not sure which section in CHANGES.txt should this belongs to when I tried to commit this yesterday. I have committed this. Thanks, Kan. > Add more access token tests > --- > > Key: HDFS-409 > URL: https://issues.apache.org/jira/browse/HDFS-409 > Project: Hadoop HDFS > Issue Type: Test > Components: data-node, hdfs client >Reporter: Kan Zhang >Assignee: Kan Zhang > Fix For: 0.21.0 > > Attachments: AccessTokenTestPlan.pdf, h409-23.patch, h409-25.patch, > h409-27.patch, h409-29.patch > > > Need to add more unit tests for access token -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-409) Add more access token tests
[ https://issues.apache.org/jira/browse/HDFS-409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-409: Resolution: Fixed Fix Version/s: 0.21.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) > Add more access token tests > --- > > Key: HDFS-409 > URL: https://issues.apache.org/jira/browse/HDFS-409 > Project: Hadoop HDFS > Issue Type: Test > Components: data-node, hdfs client >Reporter: Kan Zhang >Assignee: Kan Zhang > Fix For: 0.21.0 > > Attachments: AccessTokenTestPlan.pdf, h409-23.patch, h409-25.patch, > h409-27.patch, h409-29.patch > > > Need to add more unit tests for access token -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-542) Support fault injection tests in Hudson
[ https://issues.apache.org/jira/browse/HDFS-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742475#action_12742475 ] Tsz Wo (Nicholas), SZE commented on HDFS-542: - I did a little experience in HDFS-483 as following: - Submitted some new fault injection tests, which were failed by trunk, to Hudson. - Got [-1 on core tests|https://issues.apache.org/jira/browse/HDFS-483?focusedCommentId=12742196&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12742196], which was correct. - However, the [test report|http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/60/testReport/] did not show the failed tests. > Support fault injection tests in Hudson > --- > > Key: HDFS-542 > URL: https://issues.apache.org/jira/browse/HDFS-542 > Project: Hadoop HDFS > Issue Type: Task > Components: build, test >Reporter: Tsz Wo (Nicholas), SZE > > Fault injection tests are supported by the code base. However, Hudson cannot > show the failing the tests on the test reports. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-536) Support hflush at DFSClient
[ https://issues.apache.org/jira/browse/HDFS-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-536: --- Attachment: hdfsHflush1.patch The new patch incorporates review comments. > The patch adds a response.close(). Should we use closeResponder() instead of > response.close()? Or should we remove the later closeResponder() call? closeResponder() can not be used because it additionally has a join. The later closeResponder() can not be removed either because there is no guarantee that close was called when exiting the loop. > Support hflush at DFSClient > --- > > Key: HDFS-536 > URL: https://issues.apache.org/jira/browse/HDFS-536 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: hdfsHflush.patch, hdfsHflush1.patch > > > 1. Add hflush() API to DFSOutputStream and deprecate sync(); > 2. Make pipeline close wait until the ack for the last packet is received. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-536) Support hflush at DFSClient
[ https://issues.apache.org/jira/browse/HDFS-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-536: --- Status: Patch Available (was: Open) > Support hflush at DFSClient > --- > > Key: HDFS-536 > URL: https://issues.apache.org/jira/browse/HDFS-536 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: hdfsHflush.patch, hdfsHflush1.patch > > > 1. Add hflush() API to DFSOutputStream and deprecate sync(); > 2. Make pipeline close wait until the ack for the last packet is received. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-536) Support hflush at DFSClient
[ https://issues.apache.org/jira/browse/HDFS-536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-536: Hadoop Flags: [Reviewed] +1 the new patch looks good. Thanks, Hairong. > Support hflush at DFSClient > --- > > Key: HDFS-536 > URL: https://issues.apache.org/jira/browse/HDFS-536 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: hdfsHflush.patch, hdfsHflush1.patch > > > 1. Add hflush() API to DFSOutputStream and deprecate sync(); > 2. Make pipeline close wait until the ack for the last packet is received. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-543) Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append
Break FSDatasetInterface#writeToBlock() into writeToTemporary, writeToRBW, and append - Key: HDFS-543 URL: https://issues.apache.org/jira/browse/HDFS-543 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Affects Versions: Append Branch Reporter: Hairong Kuang Assignee: Hairong Kuang Fix For: Append Branch FSDatasetInterface#writeToBlock() currently allows to create/recover a temporary replica, create/recover a RBW replica, or append to a replica. The implementation of the method in FSDataset is very complicated and error prone. I'd like to break this method into 3: 1. writeToTemporary allows to create a Temporary replica or recover from a packet error for a Tempoary replica; 2. writeToRBW allows to create a RBW replica or recover from a packet error for a RBW replica; 3. append allows to append to an existing Finalized replica. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-544) Add a "rbw" sub directory to DataNode data directory
Add a "rbw" sub directory to DataNode data directory Key: HDFS-544 URL: https://issues.apache.org/jira/browse/HDFS-544 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Affects Versions: Append Branch Reporter: Hairong Kuang Assignee: Hairong Kuang Fix For: Append Branch Add a "rbw" sub directory to DataNode to persist RBW, RWR, and RUR replicas. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-532) Allow applications to know that a read request failed because block is missing
[ https://issues.apache.org/jira/browse/HDFS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742539#action_12742539 ] Hadoop QA commented on HDFS-532: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415799/BlockMissingException.patch against trunk revision 803622. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 96 javac compiler warnings (more than the trunk's current 95 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/63/console This message is automatically generated. > Allow applications to know that a read request failed because block is missing > -- > > Key: HDFS-532 > URL: https://issues.apache.org/jira/browse/HDFS-532 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: BlockMissingException.patch > > > I have an application that has intelligence to retrieve data from alternate > locations if HDFS cannot provide this data. This can happen when data in HDFS > is corrupted or the block is missing. HDFS already throws ChecksumException > if the block is corrupted and throws a generic IOException if the block is > missing. I would like HDFS to throw BlockMissingException when a read request > encounters a block that has no locations associated with it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-538) DistributedFileSystem::listStatus incorrectly returns null for empty result sets
[
https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan reassigned HDFS-538:
Assignee: Jakob Homan
> DistributedFileSystem::listStatus incorrectly returns null for empty result
> sets
>
>
> Key: HDFS-538
> URL: https://issues.apache.org/jira/browse/HDFS-538
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Jakob Homan
>Assignee: Jakob Homan
>
> Currently the listStatus method returns null if no files match the request.
> This differs from the Checksum/LocalFileSystem implementation, which returns
> an empty array, and the nontvery-explict prescription of the FileSystem
> interface: "{...@return the statuses of the files/directories in the given
> patch}}" It's better to return an empty collection than have to add extra
> null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-15) All replicas of a block end up on only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-15?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey reassigned HDFS-15: Assignee: Jitendra Nath Pandey (was: Hairong Kuang) > All replicas of a block end up on only 1 rack > - > > Key: HDFS-15 > URL: https://issues.apache.org/jira/browse/HDFS-15 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Hairong Kuang >Assignee: Jitendra Nath Pandey >Priority: Critical > > HDFS replicas placement strategy guarantees that the replicas of a block > exist on at least two racks when its replication factor is greater than one. > But fsck still reports that the replicas of some blocks end up on one rack. > The cause of the problem is that decommission and corruption handling only > check the block's replication factor but not the rack requirement. When an > over-replicated block loses a replica due to decomission, corruption, or > heartbeat lost, namenode does not take any action to guarantee that remaining > replicas are on different racks. > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-536) Support hflush at DFSClient
[ https://issues.apache.org/jira/browse/HDFS-536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742564#action_12742564 ] Hadoop QA commented on HDFS-536: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12416342/hdfsHflush1.patch against trunk revision 803622. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/64/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/64/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/64/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/64/console This message is automatically generated. > Support hflush at DFSClient > --- > > Key: HDFS-536 > URL: https://issues.apache.org/jira/browse/HDFS-536 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs client >Affects Versions: Append Branch >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: Append Branch > > Attachments: hdfsHflush.patch, hdfsHflush1.patch > > > 1. Add hflush() API to DFSOutputStream and deprecate sync(); > 2. Make pipeline close wait until the ack for the last packet is received. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-532) Allow applications to know that a read request failed because block is missing
[ https://issues.apache.org/jira/browse/HDFS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-532: -- Status: Open (was: Patch Available) Need to fix the javac warning introduced by this patch. > Allow applications to know that a read request failed because block is missing > -- > > Key: HDFS-532 > URL: https://issues.apache.org/jira/browse/HDFS-532 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: BlockMissingException.patch > > > I have an application that has intelligence to retrieve data from alternate > locations if HDFS cannot provide this data. This can happen when data in HDFS > is corrupted or the block is missing. HDFS already throws ChecksumException > if the block is corrupted and throws a generic IOException if the block is > missing. I would like HDFS to throw BlockMissingException when a read request > encounters a block that has no locations associated with it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-532) Allow applications to know that a read request failed because block is missing
[ https://issues.apache.org/jira/browse/HDFS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-532: -- Attachment: BlockMissingException2.txt Fix javac warning in earlier patch. > Allow applications to know that a read request failed because block is missing > -- > > Key: HDFS-532 > URL: https://issues.apache.org/jira/browse/HDFS-532 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: BlockMissingException.patch, BlockMissingException2.txt > > > I have an application that has intelligence to retrieve data from alternate > locations if HDFS cannot provide this data. This can happen when data in HDFS > is corrupted or the block is missing. HDFS already throws ChecksumException > if the block is corrupted and throws a generic IOException if the block is > missing. I would like HDFS to throw BlockMissingException when a read request > encounters a block that has no locations associated with it. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-532) Allow applications to know that a read request failed because block is missing
[
https://issues.apache.org/jira/browse/HDFS-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dhruba borthakur updated HDFS-532:
--
Status: Patch Available (was: Open)
{quote}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
{quote}
> Allow applications to know that a read request failed because block is missing
> --
>
> Key: HDFS-532
> URL: https://issues.apache.org/jira/browse/HDFS-532
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Fix For: 0.21.0
>
> Attachments: BlockMissingException.patch, BlockMissingException2.txt
>
>
> I have an application that has intelligence to retrieve data from alternate
> locations if HDFS cannot provide this data. This can happen when data in HDFS
> is corrupted or the block is missing. HDFS already throws ChecksumException
> if the block is corrupted and throws a generic IOException if the block is
> missing. I would like HDFS to throw BlockMissingException when a read request
> encounters a block that has no locations associated with it.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-200) In HDFS, sync() not yet guarantees data available to the new readers
[ https://issues.apache.org/jira/browse/HDFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-200: -- Attachment: fsyncConcurrentReaders14_20.txt This patch ignores deleting blocks during block report processing if the file is underConstruction. Updated test case to tets this corner case. > In HDFS, sync() not yet guarantees data available to the new readers > > > Key: HDFS-200 > URL: https://issues.apache.org/jira/browse/HDFS-200 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: dhruba borthakur >Priority: Blocker > Attachments: 4379_20081010TC3.java, fsyncConcurrentReaders.txt, > fsyncConcurrentReaders11_20.txt, fsyncConcurrentReaders12_20.txt, > fsyncConcurrentReaders13_20.txt, fsyncConcurrentReaders14_20.txt, > fsyncConcurrentReaders3.patch, fsyncConcurrentReaders4.patch, > fsyncConcurrentReaders5.txt, fsyncConcurrentReaders6.patch, > fsyncConcurrentReaders9.patch, > hadoop-stack-namenode-aa0-000-12.u.powerset.com.log.gz, > hypertable-namenode.log.gz, namenode.log, namenode.log, Reader.java, > Reader.java, reopen_test.sh, ReopenProblem.java, Writer.java, Writer.java > > > In the append design doc > (https://issues.apache.org/jira/secure/attachment/12370562/Appends.doc), it > says > * A reader is guaranteed to be able to read data that was 'flushed' before > the reader opened the file > However, this feature is not yet implemented. Note that the operation > 'flushed' is now called "sync". -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
