[jira] [Updated] (HDFS-6357) SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA
[ https://issues.apache.org/jira/browse/HDFS-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-6357: -- Summary: SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA (was: SetXattr should persist rpcIDs for handling Namenode restart and HA) > SetXattr should persist rpcIDs for handling retrycache with Namenode restart > and HA > --- > > Key: HDFS-6357 > URL: https://issues.apache.org/jira/browse/HDFS-6357 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-6357.patch, HDFS-6357.patch > > > SetXattr should persist rpcIDs for handling restart Namenode and HA scenarios -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6294) Use INode IDs to avoid conflicts when a file open for write is renamed
[ https://issues.apache.org/jira/browse/HDFS-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992952#comment-13992952 ] Colin Patrick McCabe commented on HDFS-6294: Looks like I need to check that the inode object is not null before setting the path based on it. > Use INode IDs to avoid conflicts when a file open for write is renamed > -- > > Key: HDFS-6294 > URL: https://issues.apache.org/jira/browse/HDFS-6294 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.20.1 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-6294.001.patch, HDFS-6294.002.patch, > HDFS-6294.003.patch, HDFS-6294.004.patch, HDFS-6294.005.patch > > > Now that we have a unique INode ID for each INode, clients with files that > are open for write can use this unique ID rather than a file path when they > are requesting more blocks or closing the open file. This will avoid > conflicts when a file which is open for write is renamed, and another file > with that name is created. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6259) Support extended attributes via WebHDFS/HttpFS
[ https://issues.apache.org/jira/browse/HDFS-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994876#comment-13994876 ] Yi Liu commented on HDFS-6259: -- Require the bug fix in HDFS-6367. > Support extended attributes via WebHDFS/HttpFS > -- > > Key: HDFS-6259 > URL: https://issues.apache.org/jira/browse/HDFS-6259 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Yi Liu >Assignee: Yi Liu > Attachments: HDFS-6259.1.patch, HDFS-6259.patch > > > Get xattrs: > {code}curl -i > 'http://:/webhdfs/v1/?op=GETXATTRS&encoding='{code} > Get xattr: > {code}curl -i > 'http://:/webhdfs/v1/?op=GETXATTR&xattr.name=&encoding='{code} > Set xattr: > {code}curl -i -X PUT > 'http://:/webhdfs/v1/?op=SETXATTR&xattr.name=&xattr.value=&flag='{code} > Remove xattr: > {code}curl -i -X PUT > 'http://:/webhdfs/v1/?op=REMOVEXATTR&xattr.name='{code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6360) MiniDFSCluster can cause unexpected side effects due to sharing of config
Kihwal Lee created HDFS-6360: Summary: MiniDFSCluster can cause unexpected side effects due to sharing of config Key: HDFS-6360 URL: https://issues.apache.org/jira/browse/HDFS-6360 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee As noted in HDFS-6329 and HDFS-5522, certain use cases of MiniDFSCluster can result in unexpected results and falsely failing or passing unit tests. Since a {{Configuration}} object is shared for all namenode startups, the modified conf object during a NN startup is passed to the next NN startup. The effect of the modified conf propagation and subsequent modifications is different depending on whether it is a single NN cluster, HA cluster or federation cluster. It also depends on what test cases are doing with the config. For example, MiniDFSCluster#getConfiguration(int) returns the saved conf for the specified NN, but that is not actually the conf object used by the NN. It just contained the same content one time in the past and it is not guaranteed to be that way. Restarting the same NN can also cause unexpected results. The new NN will switch to the conf that was cloned & saved AFTER the last startup. The new NN will start with a changed config intentionally or unintentionally. The config variables such as {{fs.defaultFs}}, {{dfs.namenode.rpc-address}} will be implicitly set differently than the initial condition. Some test cases rely on this and others occasionally break because of this. In summary, * MiniDFSCluster does not properly isolate configs. * Many test cases happen to work most of times. Correcting MiniDFSCluster causes mass breakages of test cases and requires fixing them. * Many test cases rely on broken behavior and might pass when they should have actually failed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HDFS-5688) Wire-encription in QJM
[ https://issues.apache.org/jira/browse/HDFS-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Carlos Fernandez reopened HDFS-5688: - Why has being marked as closed? I wasn't able to run QJM + SSL > Wire-encription in QJM > -- > > Key: HDFS-5688 > URL: https://issues.apache.org/jira/browse/HDFS-5688 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, journal-node, security >Affects Versions: 2.2.0 >Reporter: Juan Carlos Fernandez >Priority: Blocker > Labels: security > Attachments: core-site.xml, hdfs-site.xml, jaas.conf, journal.xml, > namenode.xml, ssl-client.xml, ssl-server.xml > > > When HA is implemented with QJM and using kerberos, it's not possible to set > wire-encrypted data. > If it's set property hadoop.rpc.protection to something different to > authentication it doesn't work propertly, getting the error: > ERROR security.UserGroupInformation: PriviledgedActionException > as:principal@REALM (auth:KERBEROS) cause:javax.security.sasl.SaslException: > No common protection layer between client and server > With NFS as shared storage everything works like a charm -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6367) EnumSetParam$Domain#parse fails for parameter containing more than one enum.
[ https://issues.apache.org/jira/browse/HDFS-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-6367: -- Summary: EnumSetParam$Domain#parse fails for parameter containing more than one enum. (was: Domain>#parse in EnumSetParam fails for parmater containing more than one enum.) > EnumSetParam$Domain#parse fails for parameter containing more than one enum. > > > Key: HDFS-6367 > URL: https://issues.apache.org/jira/browse/HDFS-6367 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Fix For: 3.0.0 > > Attachments: HDFS-6367.patch > > > Fails because additional "," > > java.lang.IllegalArgumentException: No enum const class > org.apache.hadoop.fs.Options$Rename.,OVERWRITE > at java.lang.Enum.valueOf(Enum.java:196) > at > org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85) > at > org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.(RenameOptionSetParam.java:45) > at > org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6367) EnumSetParam$Domain#parse fails for parameter containing more than one enum.
[ https://issues.apache.org/jira/browse/HDFS-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994998#comment-13994998 ] Uma Maheswara Rao G commented on HDFS-6367: --- +1 Patch looks good to me. > EnumSetParam$Domain#parse fails for parameter containing more than one enum. > > > Key: HDFS-6367 > URL: https://issues.apache.org/jira/browse/HDFS-6367 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Fix For: 3.0.0 > > Attachments: HDFS-6367.patch > > > Fails because additional "," > > java.lang.IllegalArgumentException: No enum const class > org.apache.hadoop.fs.Options$Rename.,OVERWRITE > at java.lang.Enum.valueOf(Enum.java:196) > at > org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85) > at > org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.(RenameOptionSetParam.java:45) > at > org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6371) In HA setup, the standby NN should redirect WebHDFS write requests to the active NN
Tsz Wo Nicholas Sze created HDFS-6371: - Summary: In HA setup, the standby NN should redirect WebHDFS write requests to the active NN Key: HDFS-6371 URL: https://issues.apache.org/jira/browse/HDFS-6371 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze The current WebHDFS implementation in namenode does not check its HA state -- it does the same thing no matter it is active or standby. Suppose a http client talk to the standby NN via WebHDFS. For the read operations, there is no problem. For the write operations, if the operation requires http redirect (e.g. creating a file), it will work since the standby NN will also redirect the client to a DN. When the client connect to the DN, the DN will fulfill the request with the active NN. However, for the write operations not requiring http redirect (e.g. mkdir), the operation will fail with StandbyException since it will be executed with the standby NN. There are two solutions: # The http client could catch StandbyException and then retries with the other NN in this case. # The standby NN redirects the request to the active NN. The second solution seems better since the client does not need to know both active NN and standby NN. Note that WebHdfsFileSystem is already able to handle HA failover. The JIRA is for other http clients. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995148#comment-13995148 ] Daryn Sharp commented on HDFS-6326: --- I still need to look at the latest code from [~cnauroth], but [~wheat9], the assumption in the web UI code is completely wrong and prevents any future use of any upper bits in the mask. I think avoiding the performance penalty of double rpc/http calls just to print a "+" is more important than working around a web bug. > WebHdfs ACL compatibility is broken > --- > > Key: HDFS-6326 > URL: https://issues.apache.org/jira/browse/HDFS-6326 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Chris Nauroth >Priority: Blocker > Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch > > > 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN > throws an {{IllegalArgumentException}} exception. > {code} > hadoop fs -ls webhdfs://nn/ > Found 21 items > ls: Invalid value for webhdfs parameter "op": No enum constant > org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS > [... 20 more times...] > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6294) Use INode IDs to avoid conflicts when a file open for write is renamed
[ https://issues.apache.org/jira/browse/HDFS-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-6294: --- Attachment: HDFS-6294.005.patch > Use INode IDs to avoid conflicts when a file open for write is renamed > -- > > Key: HDFS-6294 > URL: https://issues.apache.org/jira/browse/HDFS-6294 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.20.1 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-6294.001.patch, HDFS-6294.002.patch, > HDFS-6294.003.patch, HDFS-6294.004.patch, HDFS-6294.005.patch > > > Now that we have a unique INode ID for each INode, clients with files that > are open for write can use this unique ID rather than a file path when they > are requesting more blocks or closing the open file. This will avoid > conflicts when a file which is open for write is renamed, and another file > with that name is created. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6346) Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call
[ https://issues.apache.org/jira/browse/HDFS-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992960#comment-13992960 ] Uma Maheswara Rao G commented on HDFS-6346: --- Patch looks great! I have few initial comments below: {code} if (removedXAttr != null) { fsImage.getEditLog().logRemoveXAttr(src, removedXAttr); } {code} Do you think, we need warn/log to the user that xattr attempting to remove does not exist? {code} XAttr unprotectedRemoveXAttr(String src, XAttr xAttr) throws IOException { {code} Need format {code} XAttrStorage.updateINodeXAttrs(inode, newXAttrs, snapshotId); return existingXAttrs.size() == newXAttrs.size() ? null : xAttr; {code} Pls remove empty line between When existing and updated xattrs equal then we need not even call updateINodeXAttr so code can be changed from {code} {code} to {code} if (existingXAttrs.size() != newXAttrs.size()) { XAttrStorage.updateINodeXAttrs(inode, newXAttrs, snapshotId); return xAttr; } return null; {code} ? I will continue my review tomorrow on this and post if there are any comments. > Optimize OP_SET_XATTRS by persisting single Xattr entry per > setXattr/removeXattr api call > - > > Key: HDFS-6346 > URL: https://issues.apache.org/jira/browse/HDFS-6346 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Yi Liu > Attachments: HDFS-6346.patch, editsStored > > > When a new xattrs set on an Inode, it may add this with OP_SET_XATTRS and > let's say [user.name1:value1] > On a next call if we set another xattrs, then it may store along with older > existing xattrs again. It may be like [user.name1:value1, user.name2:value2] > So, on adding more xattrs on same Inode, that list may grow and we keep store > the entries of already existing name, value fairs. > Right now we defaulted the max Xattrs on an Inode to 32 and configured. If > user modified it to much larger value and start setting xattrs, then edits > loading may create issue like my above example. > But I didn't refer any usecase of having large number of xattrs, this is just > the assumption to consider a case. My biggest doubt is whether we will have > such real usecases to have huge xattrs on a single INode. > So, here is a thought on having OP_SET_XATTR for each setXAttr operation to > be logged, When we replay them we need to consolidate. This is some initial > thought we can think more if others also feel we need to consider this case > to handle. > Otherwise we endup storing Xattrs entries in editlog file as n(n+1)/2 where n > is number xattrs for a file/dir. This may be issue only when we have large > number configured max xattrs for inode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6366) FsImage loading failed with RemoveXattr op
[ https://issues.apache.org/jira/browse/HDFS-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994179#comment-13994179 ] Yi Liu commented on HDFS-6366: -- Right, Uma, in patch of HDFS-6346, we missed a break for OP_REMOVE_XATTR... > FsImage loading failed with RemoveXattr op > -- > > Key: HDFS-6366 > URL: https://issues.apache.org/jira/browse/HDFS-6366 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS XAttrs (HDFS-2006) > > Attachments: HDFS-6366.patch > > > Should break after the removeXattr op loading. > Otherwise fsimage loading will fail: > java.io.IOException: Invalid operation read OP_REMOVE_XATTR > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:816) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:227) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:136) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:902) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:651) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:479) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:535) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:694) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:679) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1332) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:973) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:854) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:700) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:373) > at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:354) > at > org.apache.hadoop.hdfs.server.namenode.FSXAttrBaseTest.initCluster(FSXAttrBaseTest.java:400) > at > org.apache.hadoop.hdfs.server.namenode.FSXAttrBaseTest.restart(FSXAttrBaseTest.java:418) > at > org.apache.hadoop.hdfs.server.namenode.FSXAttrBaseTest.testCreateXAttr(FSXAttrBaseTest.java:121) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6357) SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA
[ https://issues.apache.org/jira/browse/HDFS-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994196#comment-13994196 ] Yi Liu commented on HDFS-6357: -- +1, the patch looks good to me. Thanks Uma. > SetXattr should persist rpcIDs for handling retrycache with Namenode restart > and HA > --- > > Key: HDFS-6357 > URL: https://issues.apache.org/jira/browse/HDFS-6357 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS XAttrs (HDFS-2006) > > Attachments: HDFS-6357.patch, HDFS-6357.patch, HDFS-6357.patch > > > SetXattr should persist rpcIDs for handling restart Namenode and HA scenarios -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995278#comment-13995278 ] Haohui Mai commented on HDFS-6326: -- bq. the assumption in the web UI code is completely wrong and prevents any future use of any upper bits in the mask. The point is to demonstrate how other clients of webhdfs can potentially depend on this behavior. Though the bug is within our reach and can be fixed by ourselves, the webhdfs protocol is intended to be implemented by 3rd-party clients which are totally out of our control. I have no problem to declare this is a wrong assumption but the sad fact is that 3rd-party clients might depend on it. bq. I think avoiding the performance penalty of double rpc/http calls just to print a "+" is more important than working around a web bug. Having the bit obviously makes implementing ls / web ui much easier, but it comes with a performance cost on the critical path. We've implemented this approach before and reverted it to the current state, as Chris mentioned in earlier comments. {{ls}} is not in the critical path, but adding a new field into the {{FileStatus}} response is. Provided that the {{FileStatus}} field is relatively short (~200 bytes), adding a field for ACL increases the network traffic for listing files by 5%. The overhead is indeed significant as listing files is the first step of distcp which is done within a single thread. We've seen use cases that this step can take more than 7 hours. Again having an ACL bit makes things easier to implement, but failing to justify the performance concerns leads Chris and I to decide to keep ACL out of the critical path, and to move the complexity to ls / web UI. Maybe I don't understand what you're proposing, can you propose your changes more concretely, just like what Chris did in https://issues.apache.org/jira/browse/HDFS-6326?focusedCommentId=13991788&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13991788 > WebHdfs ACL compatibility is broken > --- > > Key: HDFS-6326 > URL: https://issues.apache.org/jira/browse/HDFS-6326 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Chris Nauroth >Priority: Blocker > Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch > > > 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN > throws an {{IllegalArgumentException}} exception. > {code} > hadoop fs -ls webhdfs://nn/ > Found 21 items > ls: Invalid value for webhdfs parameter "op": No enum constant > org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS > [... 20 more times...] > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6230) Expose upgrade status through NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6230: - Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed the patch to trunk and branch-2. Thanks [~mitdesai] for the contribution. > Expose upgrade status through NameNode web UI > - > > Key: HDFS-6230 > URL: https://issues.apache.org/jira/browse/HDFS-6230 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Arpit Agarwal >Assignee: Mit Desai > Fix For: 2.5.0 > > Attachments: HDFS-6230-NoUpgradesInProgress.png, > HDFS-6230-UpgradeInProgress.jpg, HDFS-6230.patch, HDFS-6230.patch > > > The NameNode web UI does not show upgrade information anymore. Hadoop 2.0 > also does not have the _hadoop dfsadmin -upgradeProgress_ command to check > the upgrade status. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
[ https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995353#comment-13995353 ] Colin Patrick McCabe commented on HDFS-6361: Some background would probably be helpful here. On Linux, user IDs (UIDs) are 32 bits. This convention goes all the way down into the kernel. Typically they are treated as unsigned 32-bit numbers. So you'll find code like this in {{bits/typesizes.h}}: {code} bits/typesizes.h:#define __UID_T_TYPE __U32_TYPE {code} Java, of course, does not have unsigned numbers. So we have to represent numbers like 4294967295 as -1 in Java. {code} However, the effect of returning -2 from getUid and getGid for + * nfsnobody is yet to be understood {code} What is it that we have yet to understand? We better figure it out before we commit this code. {code} + private static Integer parseId(final String idStr) { +String tmpStr = idStr; +if (idStr.equals("4294967294")) { + tmpStr = "-2"; +} +return Integer.valueOf(tmpStr); + } {code} I don't see a reason to special-case -2. We should treat all large integers > 0x7fff as negatives. If Java had unsigned numbers, we could just use an unsigned 32 bit int... but it doesn't, so we can just use a signed int. Basically, for the purposes of user IDs, -2 and 4294967294 are the same... at least on Linux. {code} Another option to the patch I provided is to use something like a 64 bit integer for ID, but the would be wide scope, and I suspect it's going to be an overkill. {code} The Linux kernel itself doesn't use 64-bit numbers for UIDs. I'm not aware of any kernels which do. So I don't see a benefit to this. > TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id > - > > Key: HDFS-6361 > URL: https://issues.apache.org/jira/browse/HDFS-6361 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6361.001.patch > > > The following error happens pretty often: > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting > Failing for the past 1 build (Since Unstable#61 ) > Took 0.1 sec. > add description > Error Message > For input string: "4294967294" > Stacktrace > java.lang.NumberFormatException: For input string: "4294967294" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:495) > at java.lang.Integer.valueOf(Integer.java:582) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188) > at org.apache.hadoop.nfs.nfs3.IdUserGroup.(IdUserGroup.java:60) > at > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71) > Standard Output > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.nfs.nfs3.IdUserGroup). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995256#comment-13995256 ] Daryn Sharp commented on HDFS-6326: --- I was going to comment on the lack of bitwise and after the shift. :) Other comments/questions: * Bits 10 & 11 are for set-uid/gid so we should probably use bit 12 to be safe. * Since the acl bit is intended to be invisible to the client, shouldn't it be ignored in {{FSPermission#equals}}? * Same for {{FsPermission#toString}}, should probably be left to FsShell or others to add the "+". * In {{AclCommands#processPath}}, is it guaranteed (probably yes?) that the owner/group/etc will be identical in the acl vs. file status? If no, maybe the first line should be changed to call getAclStatus if the bit is set, and most of the subsequent code left alone? * Should we avoid changing {{PBHelper. convert(FsPermission)}} and just let {{FsAclPermission#toShort}} serialize the bit? * Does {{PBHelper.convert(FsPermissionProto)}} need to be changed? Assuming it's only used to decode incoming permissions in a rpc call where we don't care about the bit? > WebHdfs ACL compatibility is broken > --- > > Key: HDFS-6326 > URL: https://issues.apache.org/jira/browse/HDFS-6326 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Chris Nauroth >Priority: Blocker > Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch > > > 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN > throws an {{IllegalArgumentException}} exception. > {code} > hadoop fs -ls webhdfs://nn/ > Found 21 items > ls: Invalid value for webhdfs parameter "op": No enum constant > org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS > [... 20 more times...] > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6355) Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner
[ https://issues.apache.org/jira/browse/HDFS-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995358#comment-13995358 ] Colin Patrick McCabe commented on HDFS-6355: bq. Here since we already use monotonicNow(), which is relative to jvm startup, I don't think we will get a negative value here and hence no need to take a max(..). The issue is that {{currentPeriodStart}} is not going to increase over the course of a scan, but {{Time.monotonicNow}} will. So eventually, that quantity could become negative. This is certainly a corner case (a very, very slow scan?) but it seems better to handle it. bq. +1 thanks, will commit later today if there's no more comments > Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner > > > Key: HDFS-6355 > URL: https://issues.apache.org/jira/browse/HDFS-6355 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-6355.001.patch > > > BlockPoolSliceScanner uses {{Time.now}} to calculate an interval. But this > is incorrect, since if the wall-clock time changes, we will end up setting > the scan periods to a shorter or longer time than we configured. > There is also a case where we may divide by zero if we get unlucky, because > we calculate an interval and divide by it, without checking whether the > interval is 0 milliseconds. This would produce an {{ArithmeticException}} > since we are using longs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4437) Clients should not retain leases on moved files
[ https://issues.apache.org/jira/browse/HDFS-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995374#comment-13995374 ] Colin Patrick McCabe commented on HDFS-4437: Marking this as a dupe of HDFS-6294, which addressed the issues we had with moving files that were open for write. Feel free to reopen if there's something more we should do here. > Clients should not retain leases on moved files > --- > > Key: HDFS-4437 > URL: https://issues.apache.org/jira/browse/HDFS-4437 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > Moving open files and/or parent directories of open files will rewrite the > leases to the new path. The client is not notified so future stream > operations will fail. However, as long as the client keeps its lease active > it will have a "lock" on the file in its new location. This is not good for > a daemon. > Leases should be released after the file is moved. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995335#comment-13995335 ] Haohui Mai commented on HDFS-6293: -- The patch looks good to me. One nit is that can you please mark the class that are brought back deprecated? +1 once it is addressed. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Haohui Mai >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6294) Use INode IDs to avoid conflicts when a file open for write is renamed
[ https://issues.apache.org/jira/browse/HDFS-6294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995399#comment-13995399 ] Hudson commented on HDFS-6294: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5603 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5603/]) HDFS-6294. Use INode IDs to avoid conflicts when a file open for write is renamed (cmccabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593634) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/LeaseRenewer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSClientAdapter.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLease.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRenewer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPersistBlocks.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestRenameWhileOpen.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetrepDecreasing.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSetrepIncreasing.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java > Use INode IDs to avoid conflicts when a file open for write is renamed > -- > > Key: HDFS-6294 > URL: https://issues.apache.org/jira/browse/HDFS-6294 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.20.1 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.5.0 > > Attachments: HDFS-6294.001.patch, HDFS-6294.002.patch, > HDFS-6294.003.patch, HDFS-6294.004.patch, HDFS-6294.005.patch, > HDFS-6294.005.patch > > > Now that we have a unique INode ID for each INode, clients with files that > are open for write can use this unique ID rather than a file path when they > are requesting more blocks or closing the open file. This will avoid > conflicts when a file which is open for write is renamed, and another file > with that name is created. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6351) Command "hdfs dfs -rm -r" can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995296#comment-13995296 ] Andrew Wang commented on HDFS-6351: --- I think these are both known flakes, and the fact that identical patches have produced different test runs also supports this hypothesis. I'll commit this shortly. > Command "hdfs dfs -rm -r" can't remove empty directory > -- > > Key: HDFS-6351 > URL: https://issues.apache.org/jira/browse/HDFS-6351 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6351.001.patch, HDFS-6351.002.patch, > HDFS-6351.002.patch > > > This JIRA is fo the rmr part of the issue reported in HDFS-6165. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995430#comment-13995430 ] Chris Nauroth commented on HDFS-6326: - Daryn, thank you for looking. bq. I was going to comment on the lack of bitwise and after the shift. :-) Darn, there is no hiding my shame. :-) bq. Bits 10 & 11 are for set-uid/gid so we should probably use bit 12 to be safe. Yes, we can use bit 12. bq. Since the acl bit is intended to be invisible to the client, shouldn't it be ignored in {{FSPermission#equals}}? OK, let's do that. I'll re-raise my concern that this whole business of hiding the bit causes confusion in the client-side API. Now, we'll have a situation where 2 instances can appear equal even though one has the ACL bit and the other doesn't. I suppose it's best that we're at least consistent in ignoring it, so let's remove it from {{equals}}. bq. Same for {{FsPermission#toString}}, should probably be left to {{FsShell}} or others to add the "+". Yes, same as above. bq. In {{AclCommands#processPath}}, is it guaranteed (probably yes?) that the owner/group/etc will be identical in the acl vs. file status? Yes, this is guaranteed, barring race conditions involving a concurrent process running chown interleaved between the {{getFileInfo}} and the {{getAclStatus}} RPCs. bq. Should we avoid changing {{PBHelper. convert(FsPermission)}} and just let {{FsAclPermission#toShort}} serialize the bit? Yes, we can do that. bq. Does {{PBHelper.convert(FsPermissionProto)}} need to be changed? Assuming it's only used to decode incoming permissions in a rpc call where we don't care about the bit? This change is required so that the shell (and others) can get a {{true}} return from {{FsPermission#getAclBit}} via the {{FsAclPermission}} subclass. The patch is hiding the bit from the 16-bit representation seen by callers of {{FsPermission#toShort}}, but we need to preserve the information somehow. The PB conversion layer seems to be the only remaining choice, but let me know if you had another idea in mind. > WebHdfs ACL compatibility is broken > --- > > Key: HDFS-6326 > URL: https://issues.apache.org/jira/browse/HDFS-6326 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Chris Nauroth >Priority: Blocker > Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch > > > 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN > throws an {{IllegalArgumentException}} exception. > {code} > hadoop fs -ls webhdfs://nn/ > Found 21 items > ls: Invalid value for webhdfs parameter "op": No enum constant > org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS > [... 20 more times...] > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reopened HDFS-6134: -- [cross-posting with HADOOP-10150] Reopening HDFS-6134 After some offline discussions with Yi, Tianyou, ATM, Todd, Andrew and Charles we think is makes more sense to implement encryption for HDFS directly into the DistributedFileSystem client and to use CryptoFileSystem support encryption for FileSystems that donât support native encryption. The reasons for this change of course are: * If we want to we add support for HDFS transparent compression, the compression should be done before the encryption (implying less entropy). If compression is to be handled by HDFS DistributedFileSystem, then the encryption has to be handled afterwards (in the write path). * The proposed CryptoSupport abstraction significantly complicates the implementation of CryptoFileSystem and the wiring in HDFS FileSystem client. * Building it directly into HDFS FileSystem client may allow us to avoid an extra copy of data. Because of this, the idea is now: * A common set of Crypto Input/Output streams. They would be used by CryptoFileSystem, HDFS encryption, MapReduce intermediate data and spills. Note we cannot use the JDK Cipher Input/Output streams directly because we need to support the additional interfaces that the Hadoop FileSystem streams implement (Seekable, PositionedReadable, ByteBufferReadable, HasFileDescriptor, CanSetDropBehind, CanSetReadahead, HasEnhancedByteBufferAccess, Syncable, CanSetDropBehind). * CryptoFileSystem. To support encryption in arbitrary FileSystems. * HDFS client encryption. To support transparent HDFS encryption. Both CryptoFilesystem and HDFS client encryption implementations would be built using the Crypto Input/Output streams, xAttributes and KeyProvider API. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6351) Command "hdfs dfs -rm -r" can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6351: -- Resolution: Fixed Fix Version/s: 2.5.0 Target Version/s: 2.5.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Thanks again Yongjun. > Command "hdfs dfs -rm -r" can't remove empty directory > -- > > Key: HDFS-6351 > URL: https://issues.apache.org/jira/browse/HDFS-6351 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Fix For: 2.5.0 > > Attachments: HDFS-6351.001.patch, HDFS-6351.002.patch, > HDFS-6351.002.patch > > > This JIRA is fo the rmr part of the issue reported in HDFS-6165. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6230) Expose upgrade status through NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995400#comment-13995400 ] Hudson commented on HDFS-6230: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5603 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5603/]) HDFS-6230. Expose upgrade status through NameNode web UI. Contributed by Mit Desai. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594040) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.html > Expose upgrade status through NameNode web UI > - > > Key: HDFS-6230 > URL: https://issues.apache.org/jira/browse/HDFS-6230 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Arpit Agarwal >Assignee: Mit Desai > Fix For: 2.5.0 > > Attachments: HDFS-6230-NoUpgradesInProgress.png, > HDFS-6230-UpgradeInProgress.jpg, HDFS-6230.patch, HDFS-6230.patch > > > The NameNode web UI does not show upgrade information anymore. Hadoop 2.0 > also does not have the _hadoop dfsadmin -upgradeProgress_ command to check > the upgrade status. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Attachment: HDFS-6325.patch > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, > appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6372) Handle setXattr rpcIDs for OfflineEditsViewer
[ https://issues.apache.org/jira/browse/HDFS-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-6372: -- Attachment: editsStored HDFS-6372.patch Attaching the patch for fixing the issues. I took advantage of this JIRA to fix the retrycachewithHA testcase. > Handle setXattr rpcIDs for OfflineEditsViewer > - > > Key: HDFS-6372 > URL: https://issues.apache.org/jira/browse/HDFS-6372 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-6372.patch, editsStored > > > toXml and fromXml methods in SetXattrOp should store and read the rpcids. > This JIRA can also fix the failures of > TestOfflineEditsViewer > TestRetryCacheWithHA -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5522) Datanode disk error check may be incorrectly skipped
[ https://issues.apache.org/jira/browse/HDFS-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995476#comment-13995476 ] Kihwal Lee commented on HDFS-5522: -- + 1 lgtm > Datanode disk error check may be incorrectly skipped > > > Key: HDFS-5522 > URL: https://issues.apache.org/jira/browse/HDFS-5522 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.9, 2.2.0 >Reporter: Kihwal Lee >Assignee: Rushabh S Shah > Attachments: HDFS-5522-v2.patch, HDFS-5522-v3.patch, HDFS-5522.patch > > > After HDFS-4581 and HDFS-4699, {{checkDiskError()}} is not called when > network errors occur during processing data node requests. This appears to > create problems when a disk is having problems, but not failing I/O soon. > If I/O hangs for a long time, network read/write may timeout first and the > peer may close the connection. Although the error was caused by a faulty > local disk, disk check is not being carried out in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6362) InvalidateBlocks is inconsistent in usage of DatanodeUuid and StorageID
[ https://issues.apache.org/jira/browse/HDFS-6362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993497#comment-13993497 ] Hadoop QA commented on HDFS-6362: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644077/HDFS-6362.03.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6866//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6866//console This message is automatically generated. > InvalidateBlocks is inconsistent in usage of DatanodeUuid and StorageID > --- > > Key: HDFS-6362 > URL: https://issues.apache.org/jira/browse/HDFS-6362 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Blocker > Attachments: HDFS-6362.01.patch, HDFS-6362.02.patch, > HDFS-6362.03.patch > > > {{InvalidateBlocks}} must consistently use datanodeUuid as the key. e.g. add > and remove functions use datanode UUID and storage ID. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995227#comment-13995227 ] Chris Nauroth commented on HDFS-6326: - bq. ...the assumption in the web UI code is completely wrong and prevents any future use of any upper bits in the mask. +1 I'll provide a new version of the patch that includes a fix for the web UI bug. BTW, my v3 patch has a small bug in the bitwise operations to check the ACL bit. I'll fix that in the next patch too. > WebHdfs ACL compatibility is broken > --- > > Key: HDFS-6326 > URL: https://issues.apache.org/jira/browse/HDFS-6326 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Chris Nauroth >Priority: Blocker > Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch > > > 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN > throws an {{IllegalArgumentException}} exception. > {code} > hadoop fs -ls webhdfs://nn/ > Found 21 items > ls: Invalid value for webhdfs parameter "op": No enum constant > org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS > [... 20 more times...] > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6351) Command "hdfs dfs -rm -r" can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995348#comment-13995348 ] Yongjun Zhang commented on HDFS-6351: - Thanks a lot Andrew! And many thanks to the folks who helped reviewing and discussion! > Command "hdfs dfs -rm -r" can't remove empty directory > -- > > Key: HDFS-6351 > URL: https://issues.apache.org/jira/browse/HDFS-6351 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Fix For: 2.5.0 > > Attachments: HDFS-6351.001.patch, HDFS-6351.002.patch, > HDFS-6351.002.patch > > > This JIRA is fo the rmr part of the issue reported in HDFS-6165. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995480#comment-13995480 ] Akira AJISAKA commented on HDFS-6293: - The patch looks good to me. Two comments about oiv_legacy: # -maxSize and -step options will fail (originally reported by HDFS-5866) # missing '\n' in the usage (HDFS-5864) I'm thinking the main purpose of the issue is to bring back the legacy code to keep backward compatibility, so we can fix them in a separate jira. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Haohui Mai >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995588#comment-13995588 ] Owen O'Malley commented on HDFS-6134: - What are the use cases this is trying to address? What are the attacks? Do users or administrators set the encryption? Can different directories have different keys or is it one key for the entire filesystem? When you rename a directory does it need to be re-encrypted? How are backups handled? Does it require the encryption key? What is the performance impact on distcp when not using native libraries? For release in the Hadoop 2.x line, you need to preserve both forward and backwards wire compatibility. How do you plan to address that? It seems that the additional datanode and client complexity is prohibitive. Making changes to the HDFS write and read pipeline is extremely touchy. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995650#comment-13995650 ] Hadoop QA commented on HDFS-6326: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644230/HDFS-6326.3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6875//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6875//console This message is automatically generated. > WebHdfs ACL compatibility is broken > --- > > Key: HDFS-6326 > URL: https://issues.apache.org/jira/browse/HDFS-6326 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Chris Nauroth >Priority: Blocker > Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch > > > 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN > throws an {{IllegalArgumentException}} exception. > {code} > hadoop fs -ls webhdfs://nn/ > Found 21 items > ls: Invalid value for webhdfs parameter "op": No enum constant > org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS > [... 20 more times...] > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6375) Listing extended attributes with the search permission
Andrew Wang created HDFS-6375: - Summary: Listing extended attributes with the search permission Key: HDFS-6375 URL: https://issues.apache.org/jira/browse/HDFS-6375 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Andrew Wang >From the attr(5) manpage: {noformat} Users with search access to a file or directory may retrieve a list of attribute names defined for that file or directory. {noformat} This is like doing {{getfattr}} without the {{-d}} flag, which we currently don't support. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6375) Listing extended attributes with the search permission
[ https://issues.apache.org/jira/browse/HDFS-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995652#comment-13995652 ] Andrew Wang commented on HDFS-6375: --- We don't necessarily need to add full-on support, but we should at least future-proof the getXAttr API with a flag EnumSet. > Listing extended attributes with the search permission > -- > > Key: HDFS-6375 > URL: https://issues.apache.org/jira/browse/HDFS-6375 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Andrew Wang > > From the attr(5) manpage: > {noformat} >Users with search access to a file or directory may retrieve a list of >attribute names defined for that file or directory. > {noformat} > This is like doing {{getfattr}} without the {{-d}} flag, which we currently > don't support. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6193) HftpFileSystem open should throw FileNotFoundException for non-existing paths
[ https://issues.apache.org/jira/browse/HDFS-6193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6193: - Priority: Major (was: Blocker) > HftpFileSystem open should throw FileNotFoundException for non-existing paths > - > > Key: HDFS-6193 > URL: https://issues.apache.org/jira/browse/HDFS-6193 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Gera Shegalov >Assignee: Gera Shegalov > Attachments: HDFS-6193-branch-2.4.0.v01.patch, > HDFS-6193-branch-2.4.v02.patch > > > WebHdfsFileSystem.open and HftpFileSystem.open incorrectly handles > non-existing paths. > - 'open', does not really open anything, i.e., it does not contact the > server, and therefore cannot discover FileNotFound, it's deferred until next > read. It's counterintuitive and not how local FS or HDFS work. In POSIX you > get ENOENT on open. > [LzoInputFormat.getSplits|https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/input/LzoInputFormat.java] > is an example of the code that's broken because of this. > - On the server side, FileDataServlet incorrectly sends SC_BAD_REQUEST > instead of SC_NOT_FOUND for non-exitsing paths -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6293: - Attachment: HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch The new patch adds the old OIV back in addition to the previous changes. It can be used by "hdfs oiv_legacy" command. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Haohui Mai >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995302#comment-13995302 ] Andrew Wang commented on HDFS-6283: --- Hi [~hitliuyi], would it help if I picked this one up? I think this is one of the last things we need to get done before calling a merge vote, and I think I could knock this out in a day or two. > Write end user documentation for xattrs. > > > Key: HDFS-6283 > URL: https://issues.apache.org/jira/browse/HDFS-6283 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Reporter: Chris Nauroth >Assignee: Yi Liu > > Update the File System Shell documentation to cover the new getfattr and > setfattr commands. If warranted, consider adding a separate dedicated page > for fuller discussion of the xattrs model and how the feature works in more > detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6374) setXAttr should require the user to be the owner of the file or directory
[ https://issues.apache.org/jira/browse/HDFS-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb reassigned HDFS-6374: -- Assignee: Charles Lamb > setXAttr should require the user to be the owner of the file or directory > - > > Key: HDFS-6374 > URL: https://issues.apache.org/jira/browse/HDFS-6374 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Andrew Wang >Assignee: Charles Lamb > > From the attr(5) manpage: > {noformat} >For this reason, extended user attributes are only allowed for regular >files and directories, and access to extended user attributes is >restricted to the owner and to users with appropriate capabilities for >directories with the sticky bit set (see the chmod(1) manual page for >an explanation of Sticky Directories). > {noformat} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6351) Command "hdfs dfs -rm -r" can't remove empty directory
[ https://issues.apache.org/jira/browse/HDFS-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993907#comment-13993907 ] Hadoop QA commented on HDFS-6351: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644120/HDFS-6351.002.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.datanode.fsdataset.TestAvailableSpaceVolumeChoosingPolicy org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6867//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6867//console This message is automatically generated. > Command "hdfs dfs -rm -r" can't remove empty directory > -- > > Key: HDFS-6351 > URL: https://issues.apache.org/jira/browse/HDFS-6351 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6351.001.patch, HDFS-6351.002.patch, > HDFS-6351.002.patch > > > This JIRA is fo the rmr part of the issue reported in HDFS-6165. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5522) Datanode disk error check may be incorrectly skipped
[ https://issues.apache.org/jira/browse/HDFS-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5522: - Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for working on this, Rushabh. I've committed this to branch-2 and trunk. This feature makes disk check asynchronous, so handlers(DataXceiver) won't get blocked while checking disks. > Datanode disk error check may be incorrectly skipped > > > Key: HDFS-5522 > URL: https://issues.apache.org/jira/browse/HDFS-5522 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.9, 2.2.0 >Reporter: Kihwal Lee >Assignee: Rushabh S Shah > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-5522-v2.patch, HDFS-5522-v3.patch, HDFS-5522.patch > > > After HDFS-4581 and HDFS-4699, {{checkDiskError()}} is not called when > network errors occur during processing data node requests. This appears to > create problems when a disk is having problems, but not failing I/O soon. > If I/O hangs for a long time, network read/write may timeout first and the > peer may close the connection. Although the error was caused by a faulty > local disk, disk check is not being carried out in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6376) Distcp data between two HA clusters requires another configuration
[ https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Marion updated HDFS-6376: -- Attachment: HDFS-6376-patch-1.patch Patch 1 adds a new hdfs-site property which allows the user to specify which nameservices are *not* part of the cluster. This allows configuration information to be supplied for other clusters so that clients can resolve their locations. > Distcp data between two HA clusters requires another configuration > -- > > Key: HDFS-6376 > URL: https://issues.apache.org/jira/browse/HDFS-6376 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 2.3.0, 2.4.0 > Environment: CDH 5.0 (Apache 2.3.0 + some patches) >Reporter: Dave Marion > Attachments: HDFS-6376-patch-1.patch > > > User has to create a third set of configuration files for distcp when > transferring data between two HA clusters. > Consider the scenario in [1]. You cannot put all of the required properties > in core-site.xml and hdfs-site.xml for the client to resolve the location of > both active namenodes. If you do, then the datanodes from cluster A may join > cluster B. I can not find a configuration option that tells the datanodes to > federate blocks for only one of the clusters in the configuration. > [1] > http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6186) Pause deletion of blocks when the namenode starts up
[ https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6186: Status: Patch Available (was: Open) > Pause deletion of blocks when the namenode starts up > > > Key: HDFS-6186 > URL: https://issues.apache.org/jira/browse/HDFS-6186 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch > > > HDFS namenode can delete blocks very quickly, given the deletion happens as a > parallel operation spread across many datanodes. One of the frequent > anxieties I see is that a lot of data can be deleted very quickly, when a > cluster is brought up, especially when one of the storage directories has > failed and namenode metadata was copied from another storage. Copying wrong > metadata would results in some of the newer files (if old metadata was > copied) being deleted along with their blocks. > HDFS-5986 now captures the number of pending deletion block on namenode webUI > and JMX. I propose pausing deletion of blocks for a configured period of time > (default 1 hour?) after namenode comes out of safemode. This will give enough > time for the administrator to notice large number of pending deletion blocks > and take corrective action. > Thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6358) WebHdfs DN's DFSClient should not use a retry policy
[ https://issues.apache.org/jira/browse/HDFS-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-6358: -- Target Version/s: 2.5.0 > WebHdfs DN's DFSClient should not use a retry policy > > > Key: HDFS-6358 > URL: https://issues.apache.org/jira/browse/HDFS-6358 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > DFSClient retries on the DN are useless. The webhdfs client is going to > timeout before the retries complete. The DFSClient will also continue to run > until it timeouts. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6186) Pause deletion of blocks when the namenode starts up
[ https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6186: Attachment: HDFS-6186.002.patch Thanks for the comments, Ming and Suresh! After an offline discussion with [~sureshms], looks like a simplified mechanism here can be: to delay the block deletion for a specific period of time after NN startup, no matter the invalid block is caused by normal file deletion or unknown block in block report. Update the patch to implement the idea. The patch adds a new configuration property "dfs.block.pending.invalidation.ms" to control the delaying period, and simply handles all the delaying logic in InvalidateBlocks. > Pause deletion of blocks when the namenode starts up > > > Key: HDFS-6186 > URL: https://issues.apache.org/jira/browse/HDFS-6186 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch > > > HDFS namenode can delete blocks very quickly, given the deletion happens as a > parallel operation spread across many datanodes. One of the frequent > anxieties I see is that a lot of data can be deleted very quickly, when a > cluster is brought up, especially when one of the storage directories has > failed and namenode metadata was copied from another storage. Copying wrong > metadata would results in some of the newer files (if old metadata was > copied) being deleted along with their blocks. > HDFS-5986 now captures the number of pending deletion block on namenode webUI > and JMX. I propose pausing deletion of blocks for a configured period of time > (default 1 hour?) after namenode comes out of safemode. This will give enough > time for the administrator to notice large number of pending deletion blocks > and take corrective action. > Thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995723#comment-13995723 ] Chris Nauroth commented on HDFS-6326: - bq. Should we avoid changing PBHelper. convert(FsPermission) and just let FsAclPermission#toShort serialize the bit? Actually, I take it back. We can't do this part. The point of {{FsAclPermission}} was to hide the ACL bit from being visible in the bit representation. If the subclass just overrides {{toShort}} to do this, then it defeats the purpose. I'll keep this logic in {{PBHelper}}. > WebHdfs ACL compatibility is broken > --- > > Key: HDFS-6326 > URL: https://issues.apache.org/jira/browse/HDFS-6326 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Chris Nauroth >Priority: Blocker > Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch > > > 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN > throws an {{IllegalArgumentException}} exception. > {code} > hadoop fs -ls webhdfs://nn/ > Found 21 items > ls: Invalid value for webhdfs parameter "op": No enum constant > org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS > [... 20 more times...] > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6341) Clean up audit logging in FSNamesystem xattr methods
[ https://issues.apache.org/jira/browse/HDFS-6341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-6341. --- Resolution: Not a Problem I looked at this code again, and decided that the audit logging pattern was reasonable. Closing as not a problem. > Clean up audit logging in FSNamesystem xattr methods > > > Key: HDFS-6341 > URL: https://issues.apache.org/jira/browse/HDFS-6341 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Minor > > We call logAuditEvent multiple times in each RPC handler, sometimes split > across different functions. It'd be cleaner if we did this once, in a > try/finally. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6326) WebHdfs ACL compatibility is broken
[ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6326: Attachment: HDFS-6326.4.patch Here is patch v4 incorporating the feedback mentioned in the last several comments. While fixing the web UI, I also noticed that the logic for printing 't' vs. 'T' for the sticky bit was incorrect. Due to the way the for loop works, this was always checking the sticky bit itself instead of the other execute bit. At the point where the check is performed, the sticky bit is always on anyway, so it always chose to print 't'. I fixed this bug too. > WebHdfs ACL compatibility is broken > --- > > Key: HDFS-6326 > URL: https://issues.apache.org/jira/browse/HDFS-6326 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Chris Nauroth >Priority: Blocker > Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch, > HDFS-6326.4.patch > > > 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN > throws an {{IllegalArgumentException}} exception. > {code} > hadoop fs -ls webhdfs://nn/ > Found 21 items > ls: Invalid value for webhdfs parameter "op": No enum constant > org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS > [... 20 more times...] > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[ https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995775#comment-13995775 ] Chris Nauroth commented on HDFS-6326: - While v4 is under review, I'll repeat my manual compatibility testing using a mix of 2.3.0 and trunk components. > WebHdfs ACL compatibility is broken > --- > > Key: HDFS-6326 > URL: https://issues.apache.org/jira/browse/HDFS-6326 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Chris Nauroth >Priority: Blocker > Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch, > HDFS-6326.4.patch > > > 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN > throws an {{IllegalArgumentException}} exception. > {code} > hadoop fs -ls webhdfs://nn/ > Found 21 items > ls: Invalid value for webhdfs parameter "op": No enum constant > org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS > [... 20 more times...] > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
[ https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995783#comment-13995783 ] Brandon Li commented on HDFS-6361: -- So sorry for the late response. Thanks [~yzhangal] and [~cmccabe] for the finding the issue and giving the background description. I agree that we should not treat -2 as a special case. Basically, user can configure anything in /etc/passwd or corresponding files on purpose or by mistake. For example, on MacOS nobody is -2 and nogroup is -1. On Linux, the system converts user id "-3" in /etc/passwd to 4294967295. NFS gateway get the uid/gid from a shell command. In the result, the id string is from uint32. To handle the id from negative number, we can use Long to parse the string and then do Long.getInt() to return the correct value. {quote}..., whether all consumers of Nfs3Util.getFileAttr() will be happy{quote} The consumers of Nfs3Util.getFileAttr() should be OK with that since it looks up the id by using the name retrieved from HDFS. Let me know if I misunderstood your question. > TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id > - > > Key: HDFS-6361 > URL: https://issues.apache.org/jira/browse/HDFS-6361 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6361.001.patch > > > The following error happens pretty often: > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting > Failing for the past 1 build (Since Unstable#61 ) > Took 0.1 sec. > add description > Error Message > For input string: "4294967294" > Stacktrace > java.lang.NumberFormatException: For input string: "4294967294" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:495) > at java.lang.Integer.valueOf(Integer.java:582) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188) > at org.apache.hadoop.nfs.nfs3.IdUserGroup.(IdUserGroup.java:60) > at > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71) > Standard Output > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.nfs.nfs3.IdUserGroup). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6373) Remove support for extended attributes on symlinks
[ https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb reassigned HDFS-6373: -- Assignee: Charles Lamb > Remove support for extended attributes on symlinks > -- > > Key: HDFS-6373 > URL: https://issues.apache.org/jira/browse/HDFS-6373 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Andrew Wang >Assignee: Charles Lamb > > Looking in the Linux source code, we see the following: > http://lxr.linux.no/linux+v3.14.3/fs/xattr.c > {code} > 60/* > 61 * In the user.* namespace, only regular files and directories > can have > 62 * extended attributes. For sticky directories, only the owner and > 63 * privileged users can write attributes. > 64 */ > {code} > We should consider removing {{XAttrFeature}} from {{INodeSymlink}}. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6376) Distcp data between two HA clusters requires another configuration
[ https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995773#comment-13995773 ] Hadoop QA commented on HDFS-6376: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644500/HDFS-6376-patch-1.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6881//console This message is automatically generated. > Distcp data between two HA clusters requires another configuration > -- > > Key: HDFS-6376 > URL: https://issues.apache.org/jira/browse/HDFS-6376 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 2.3.0, 2.4.0 > Environment: CDH 5.0 (Apache 2.3.0 + some patches) >Reporter: Dave Marion > Attachments: HDFS-6376-patch-1.patch > > > User has to create a third set of configuration files for distcp when > transferring data between two HA clusters. > Consider the scenario in [1]. You cannot put all of the required properties > in core-site.xml and hdfs-site.xml for the client to resolve the location of > both active namenodes. If you do, then the datanodes from cluster A may join > cluster B. I can not find a configuration option that tells the datanodes to > federate blocks for only one of the clusters in the configuration. > [1] > http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6377) Unify xattr name and value limits into a single limit
Andrew Wang created HDFS-6377: - Summary: Unify xattr name and value limits into a single limit Key: HDFS-6377 URL: https://issues.apache.org/jira/browse/HDFS-6377 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Andrew Wang Instead of having separate limits and config options for the size of an xattr's name and value, let's use a single limit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995828#comment-13995828 ] Andrew Purtell commented on HDFS-6134: -- {quote} Because of this, the idea is now: - A common set of Crypto Input/Output streams. [ ... ] - CryptoFileSystem. [ ... ] - HDFS client encryption. [ ... ] {quote} I would be great if the work is structured this way. A filtering CryptoFileSystem is needed for filesystem agnostic client side use cases, but e.g. if we want to push compression and encryption in HBase down into HDFS (which I think is desirable), or Hive or Pig or really any HDFS hosted Hadoop application, then doing so is far simpler if the DFS client supports transparent encryption directly. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6369) RemoteBlockReader#available() should call FSInputChecker.available()
[ https://issues.apache.org/jira/browse/HDFS-6369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995791#comment-13995791 ] Hadoop QA commented on HDFS-6369: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644364/HDFS-6369.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6878//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6878//console This message is automatically generated. > RemoteBlockReader#available() should call FSInputChecker.available() > > > Key: HDFS-6369 > URL: https://issues.apache.org/jira/browse/HDFS-6369 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ted Yu >Assignee: Chen He >Priority: Trivial > Attachments: HDFS-6369.patch > > > Currently DFSClient.TCP_WINDOW_SIZE is directly returned. > However, FSInputChecker.available(), in the superclass, may return value > lower than the constant. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6376) Distcp data between two HA clusters requires another configuration
[ https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Marion updated HDFS-6376: -- Affects Version/s: 2.4.0 Status: Patch Available (was: Open) Note that the patch was developed against CDH5 source. > Distcp data between two HA clusters requires another configuration > -- > > Key: HDFS-6376 > URL: https://issues.apache.org/jira/browse/HDFS-6376 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 2.4.0, 2.3.0 > Environment: CDH 5.0 (Apache 2.3.0 + some patches) >Reporter: Dave Marion > Attachments: HDFS-6376-patch-1.patch > > > User has to create a third set of configuration files for distcp when > transferring data between two HA clusters. > Consider the scenario in [1]. You cannot put all of the required properties > in core-site.xml and hdfs-site.xml for the client to resolve the location of > both active namenodes. If you do, then the datanodes from cluster A may join > cluster B. I can not find a configuration option that tells the datanodes to > federate blocks for only one of the clusters in the configuration. > [1] > http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6378) NFS: when portmap/rpcbind is not available, NFS registration should timeout instead of hanging
[ https://issues.apache.org/jira/browse/HDFS-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6378: - Description: When portmap/rpcbind is not available, NFS registration should timeout instead of hanging. Instead, NFS gateway should shut down automatically with proper error message. (was: NFS gateway should ) > NFS: when portmap/rpcbind is not available, NFS registration should timeout > instead of hanging > --- > > Key: HDFS-6378 > URL: https://issues.apache.org/jira/browse/HDFS-6378 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Brandon Li > > When portmap/rpcbind is not available, NFS registration should timeout > instead of hanging. Instead, NFS gateway should shut down automatically with > proper error message. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6255) fuse_dfs will not adhere to ACL permissions in some cases
[ https://issues.apache.org/jira/browse/HDFS-6255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995847#comment-13995847 ] Stephen Chu commented on HDFS-6255: --- Hi Chris and Colin, thanks for looking into this. I'll try with -oallow_other and confirm it works. > fuse_dfs will not adhere to ACL permissions in some cases > - > > Key: HDFS-6255 > URL: https://issues.apache.org/jira/browse/HDFS-6255 > Project: Hadoop HDFS > Issue Type: Bug > Components: fuse-dfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Stephen Chu >Assignee: Chris Nauroth > > As hdfs user, I created a directory /tmp/acl_dir/ and set permissions to 700. > Then I set a new acl group:jenkins:rwx on /tmp/acl_dir. > {code} > jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -getfacl /tmp/acl_dir > # file: /tmp/acl_dir > # owner: hdfs > # group: supergroup > user::rwx > group::--- > group:jenkins:rwx > mask::rwx > other::--- > {code} > Through the FsShell, the jenkins user can list /tmp/acl_dir as well as create > a file and directory inside. > {code} > [jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -touchz /tmp/acl_dir/testfile1 > [jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -mkdir /tmp/acl_dir/testdir1 > hdfs dfs -ls /tmp/acl[jenkins@hdfs-vanilla-1 ~]$ hdfs dfs -ls /tmp/acl_dir/ > Found 2 items > drwxr-xr-x - jenkins supergroup 0 2014-04-17 19:11 > /tmp/acl_dir/testdir1 > -rw-r--r-- 1 jenkins supergroup 0 2014-04-17 19:11 > /tmp/acl_dir/testfile1 > [jenkins@hdfs-vanilla-1 ~]$ > {code} > However, as the same jenkins user, when I try to cd into /tmp/acl_dir using a > fuse_dfs mount, I get permission denied. Same permission denied when I try to > create or list files. > {code} > [jenkins@hdfs-vanilla-1 tmp]$ ls -l > total 16 > drwxrwx--- 4 hdfsnobody 4096 Apr 17 19:11 acl_dir > drwx-- 2 hdfsnobody 4096 Apr 17 18:30 acl_dir_2 > drwxr-xr-x 3 mapred nobody 4096 Mar 11 03:53 mapred > drwxr-xr-x 4 jenkins nobody 4096 Apr 17 07:25 testcli > -rwx-- 1 hdfsnobody0 Apr 7 17:18 tf1 > [jenkins@hdfs-vanilla-1 tmp]$ cd acl_dir > bash: cd: acl_dir: Permission denied > [jenkins@hdfs-vanilla-1 tmp]$ touch acl_dir/testfile2 > touch: cannot touch `acl_dir/testfile2': Permission denied > [jenkins@hdfs-vanilla-1 tmp]$ mkdir acl_dir/testdir2 > mkdir: cannot create directory `acl_dir/testdir2': Permission denied > [jenkins@hdfs-vanilla-1 tmp]$ > {code} > The fuse_dfs debug output doesn't show any error for the above operations: > {code} > unique: 18, opcode: OPENDIR (27), nodeid: 2, insize: 48 >unique: 18, success, outsize: 32 > unique: 19, opcode: READDIR (28), nodeid: 2, insize: 80 > readdir[0] from 0 >unique: 19, success, outsize: 312 > unique: 20, opcode: GETATTR (3), nodeid: 2, insize: 56 > getattr /tmp >unique: 20, success, outsize: 120 > unique: 21, opcode: READDIR (28), nodeid: 2, insize: 80 >unique: 21, success, outsize: 16 > unique: 22, opcode: RELEASEDIR (29), nodeid: 2, insize: 64 >unique: 22, success, outsize: 16 > unique: 23, opcode: GETATTR (3), nodeid: 2, insize: 56 > getattr /tmp >unique: 23, success, outsize: 120 > unique: 24, opcode: GETATTR (3), nodeid: 3, insize: 56 > getattr /tmp/acl_dir >unique: 24, success, outsize: 120 > unique: 25, opcode: GETATTR (3), nodeid: 3, insize: 56 > getattr /tmp/acl_dir >unique: 25, success, outsize: 120 > unique: 26, opcode: GETATTR (3), nodeid: 3, insize: 56 > getattr /tmp/acl_dir >unique: 26, success, outsize: 120 > unique: 27, opcode: GETATTR (3), nodeid: 3, insize: 56 > getattr /tmp/acl_dir >unique: 27, success, outsize: 120 > unique: 28, opcode: GETATTR (3), nodeid: 3, insize: 56 > getattr /tmp/acl_dir >unique: 28, success, outsize: 120 > {code} > In other scenarios, ACL permissions are enforced successfully. For example, > as hdfs user I create /tmp/acl_dir_2 and set permissions to 777. I then set > the acl user:jenkins:--- on the directory. On the fuse mount, I am not able > to ls, mkdir, or touch to that directory as jenkins user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6283: -- Attachment: hdfs-6283-1.patch Patch attached. I also took the opportunity to fix some whitespace and a typo in XAttrCommands, and add the new parameters to hdfs-default.xml. > Write end user documentation for xattrs. > > > Key: HDFS-6283 > URL: https://issues.apache.org/jira/browse/HDFS-6283 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Reporter: Chris Nauroth >Assignee: Yi Liu > Attachments: hdfs-6283-1.patch > > > Update the File System Shell documentation to cover the new getfattr and > setfattr commands. If warranted, consider adding a separate dedicated page > for fuller discussion of the xattrs model and how the feature works in more > detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-6283: - Assignee: Andrew Wang (was: Yi Liu) > Write end user documentation for xattrs. > > > Key: HDFS-6283 > URL: https://issues.apache.org/jira/browse/HDFS-6283 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Reporter: Chris Nauroth >Assignee: Andrew Wang > Attachments: hdfs-6283-1.patch > > > Update the File System Shell documentation to cover the new getfattr and > setfattr commands. If warranted, consider adding a separate dedicated page > for fuller discussion of the xattrs model and how the feature works in more > detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6372) Handle setXattr rpcIDs for OfflineEditsViewer
[ https://issues.apache.org/jira/browse/HDFS-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995923#comment-13995923 ] Yi Liu commented on HDFS-6372: -- Thanks Uma, +1, the patch looks good to me. > Handle setXattr rpcIDs for OfflineEditsViewer > - > > Key: HDFS-6372 > URL: https://issues.apache.org/jira/browse/HDFS-6372 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-6372.patch, editsStored > > > toXml and fromXml methods in SetXattrOp should store and read the rpcids. > This JIRA can also fix the failures of > TestOfflineEditsViewer > TestRetryCacheWithHA -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995668#comment-13995668 ] Suresh Srinivas commented on HDFS-6293: --- I agree with [~kihwal] on deprecation. Is this also going into trunk? I think first it should be committed to trunk and then branch-2. If we are in a position to remove it from trunk later, then it can be removed later. Thoughts? +1 for the patch with Jenkins +1 (if it is going to trunk as well). > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Haohui Mai >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5682) Heterogeneous Storage phase 2 - APIs to expose Storage Types
[ https://issues.apache.org/jira/browse/HDFS-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995944#comment-13995944 ] Zesheng Wu commented on HDFS-5682: -- Thanks [~arpitagarwal], recently some of the our users have strong requirement of read/write SLA on HBase, and we notice that HDFS's heterogeneous storage feature is really interesting and helpful, thanks for your valuable work on this:) Last week I looked into HDFS-2832 and found that we need HDFS-5682 to be finished before heterogeneous storage can be used, so I asked for the progress of this issue. As you said this was deprioritized, I am wondering whether you can give a more specific plan, and I can spend some time working on some of the subtasks. > Heterogeneous Storage phase 2 - APIs to expose Storage Types > > > Key: HDFS-5682 > URL: https://issues.apache.org/jira/browse/HDFS-5682 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Phase 1 (HDFS-2832) added support to present the DataNode as a collection of > discrete storages of different types. > This Jira is to track phase 2 of the Heterogeneous Storage work which > involves exposing Storage Types to applications and adding Quota Management > support for administrators. > This phase will also include tools support for administrators/users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6373) Remove support for extended attributes on symlinks
Andrew Wang created HDFS-6373: - Summary: Remove support for extended attributes on symlinks Key: HDFS-6373 URL: https://issues.apache.org/jira/browse/HDFS-6373 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Andrew Wang Looking in the Linux source code, we see the following: http://lxr.linux.no/linux+v3.14.3/fs/xattr.c {code} 60/* 61 * In the user.* namespace, only regular files and directories can have 62 * extended attributes. For sticky directories, only the owner and 63 * privileged users can write attributes. 64 */ {code} We should consider removing {{XAttrFeature}} from {{INodeSymlink}}. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
[ https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995954#comment-13995954 ] Yongjun Zhang commented on HDFS-6361: - Hi [~cmccabe] and [~brandonli], Thanks for the comments and info. I uploaded revised version 0002, in which I used Long to parse the string as Brandon suggested to make it general, and added a couple of more entries to the unit testcase. Thanks in advance for another round of review. > TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id > - > > Key: HDFS-6361 > URL: https://issues.apache.org/jira/browse/HDFS-6361 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6361.001.patch, HDFS-6361.002.patch > > > The following error happens pretty often: > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting > Failing for the past 1 build (Since Unstable#61 ) > Took 0.1 sec. > add description > Error Message > For input string: "4294967294" > Stacktrace > java.lang.NumberFormatException: For input string: "4294967294" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:495) > at java.lang.Integer.valueOf(Integer.java:582) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188) > at org.apache.hadoop.nfs.nfs3.IdUserGroup.(IdUserGroup.java:60) > at > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71) > Standard Output > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.nfs.nfs3.IdUserGroup). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995918#comment-13995918 ] Keith Pak commented on HDFS-6325: - testFileAppend4 failure: There may be an overlap of shutdown of testAppendInsufficientLocations and startup of testCompleteOtherLeaseHoldersFile. I changed testAppendInsufficientLocations to use the field variable "cluster" instead of creating my own as a local variable and also reduced the number of DNs from 6 to 4. Otherwise, I see that testCompleteOtherLeaseHoldersFile is progressing so I think a bigger timeout may be required as stated in this JIRA: https://issues.apache.org/jira/browse/HADOOP-8596. TestBalancerWithNodeGroup: This also seems to be an intermittent issue from https://issues.apache.org/jira/browse/HDFS-6250. > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, > HDFS-6325_test.patch, appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Status: Patch Available (was: In Progress) > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, > appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6366) FsImage loading failed with RemoveXattr op
[ https://issues.apache.org/jira/browse/HDFS-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-6366. --- Resolution: Fixed Fix Version/s: HDFS XAttrs (HDFS-2006) Hadoop Flags: Reviewed Thanks for the review Yi. I have just committed this to branch! > FsImage loading failed with RemoveXattr op > -- > > Key: HDFS-6366 > URL: https://issues.apache.org/jira/browse/HDFS-6366 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS XAttrs (HDFS-2006) > > Attachments: HDFS-6366.patch > > > Should break after the removeXattr op loading. > Otherwise fsimage loading will fail: > java.io.IOException: Invalid operation read OP_REMOVE_XATTR > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:816) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:227) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:136) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:902) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:651) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:479) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:535) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:694) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:679) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1332) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6268) Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found
[ https://issues.apache.org/jira/browse/HDFS-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995942#comment-13995942 ] Andrew Wang commented on HDFS-6268: --- Manually triggered Jenkins, dunno why it didn't run. > Better sorting in NetworkTopology#pseudoSortByDistance when no local node is > found > -- > > Key: HDFS-6268 > URL: https://issues.apache.org/jira/browse/HDFS-6268 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Andrew Wang >Assignee: Andrew Wang >Priority: Minor > Attachments: hdfs-6268-1.patch, hdfs-6268-2.patch, hdfs-6268-3.patch, > hdfs-6268-4.patch > > > In NetworkTopology#pseudoSortByDistance, if no local node is found, it will > always place the first rack local node in the list in front. > This became an issue when a dataset was loaded from a single datanode. This > datanode ended up being the first replica for all the blocks in the dataset. > When running an Impala query, the non-local reads when reading past a block > boundary were all hitting this node, meaning massive load skew. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HDFS-6372) Handle setXattr rpcIDs for OfflineEditsViewer
[ https://issues.apache.org/jira/browse/HDFS-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6372 started by Uma Maheswara Rao G. > Handle setXattr rpcIDs for OfflineEditsViewer > - > > Key: HDFS-6372 > URL: https://issues.apache.org/jira/browse/HDFS-6372 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-6372.patch, editsStored > > > toXml and fromXml methods in SetXattrOp should store and read the rpcids. > This JIRA can also fix the failures of > TestOfflineEditsViewer > TestRetryCacheWithHA -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6359) WebHdfs NN servlet issues redirects in safemode or standby
Daryn Sharp created HDFS-6359: - Summary: WebHdfs NN servlet issues redirects in safemode or standby Key: HDFS-6359 URL: https://issues.apache.org/jira/browse/HDFS-6359 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Critical Webhdfs does not check for safemode or standby during issuing a redirect for open/create/checksum calls. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6377) Unify xattr name and value limits into a single limit
[ https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6377: -- Attachment: hdfs-6377-1.patch Patch attached unifying these two. I also cleaned up the xattr tests a bit. Ran TestFileContextXAttr and TestNameNodeXAttr successfully. > Unify xattr name and value limits into a single limit > - > > Key: HDFS-6377 > URL: https://issues.apache.org/jira/browse/HDFS-6377 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-6377-1.patch > > > Instead of having separate limits and config options for the size of an > xattr's name and value, let's use a single limit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HDFS-6377) Unify xattr name and value limits into a single limit
[ https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6377 started by Andrew Wang. > Unify xattr name and value limits into a single limit > - > > Key: HDFS-6377 > URL: https://issues.apache.org/jira/browse/HDFS-6377 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-6377-1.patch > > > Instead of having separate limits and config options for the size of an > xattr's name and value, let's use a single limit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6186) Pause deletion of blocks when the namenode starts up
[ https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6186: Attachment: HDFS-6186.003.patch Thanks for the review, Suresh! Update the patch to address your comments. > Pause deletion of blocks when the namenode starts up > > > Key: HDFS-6186 > URL: https://issues.apache.org/jira/browse/HDFS-6186 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch, > HDFS-6186.003.patch > > > HDFS namenode can delete blocks very quickly, given the deletion happens as a > parallel operation spread across many datanodes. One of the frequent > anxieties I see is that a lot of data can be deleted very quickly, when a > cluster is brought up, especially when one of the storage directories has > failed and namenode metadata was copied from another storage. Copying wrong > metadata would results in some of the newer files (if old metadata was > copied) being deleted along with their blocks. > HDFS-5986 now captures the number of pending deletion block on namenode webUI > and JMX. I propose pausing deletion of blocks for a configured period of time > (default 1 hour?) after namenode comes out of safemode. This will give enough > time for the administrator to notice large number of pending deletion blocks > and take corrective action. > Thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6056) Clean up NFS config settings
[ https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6056: - Hadoop Flags: Incompatible change > Clean up NFS config settings > > > Key: HDFS-6056 > URL: https://issues.apache.org/jira/browse/HDFS-6056 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Aaron T. Myers >Assignee: Brandon Li > Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, > HDFS-6056.003.patch > > > As discussed on HDFS-6050, there's a few opportunities to improve the config > settings related to NFS. This JIRA is to implement those changes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5522) Datanode disk error check may be incorrectly skipped
[ https://issues.apache.org/jira/browse/HDFS-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995980#comment-13995980 ] Hudson commented on HDFS-5522: -- FAILURE: Integrated in Hadoop-trunk-Commit #5604 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5604/]) HDFS-5522. Datanode disk error check may be incorrectly skipped. Contributed by Rushabh Shah. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594055) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDiskError.java > Datanode disk error check may be incorrectly skipped > > > Key: HDFS-5522 > URL: https://issues.apache.org/jira/browse/HDFS-5522 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.9, 2.2.0 >Reporter: Kihwal Lee >Assignee: Rushabh S Shah > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-5522-v2.patch, HDFS-5522-v3.patch, HDFS-5522.patch > > > After HDFS-4581 and HDFS-4699, {{checkDiskError()}} is not called when > network errors occur during processing data node requests. This appears to > create problems when a disk is having problems, but not failing I/O soon. > If I/O hangs for a long time, network read/write may timeout first and the > peer may close the connection. Although the error was caused by a faulty > local disk, disk check is not being carried out in this case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6376) Distcp data between two HA clusters requires another configuration
Dave Marion created HDFS-6376: - Summary: Distcp data between two HA clusters requires another configuration Key: HDFS-6376 URL: https://issues.apache.org/jira/browse/HDFS-6376 Project: Hadoop HDFS Issue Type: Bug Components: federation Affects Versions: 2.3.0 Environment: CDH 5.0 (Apache 2.3.0 + some patches) Reporter: Dave Marion User has to create a third set of configuration files for distcp when transferring data between two HA clusters. Consider the scenario in [1]. You cannot put all of the required properties in core-site.xml and hdfs-site.xml for the client to resolve the location of both active namenodes. If you do, then the datanodes from cluster A may join cluster B. I can not find a configuration option that tells the datanodes to federate blocks for only one of the clusters in the configuration. [1] http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6334) Client failover proxy provider for IP failover based NN HA
[ https://issues.apache.org/jira/browse/HDFS-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995957#comment-13995957 ] Aaron T. Myers commented on HDFS-6334: -- Patch looks great to me, Kihwal. Thanks for doing it. Two little nits: # bq. + * used, a special token handling mat be needed to make sure a token acquired s/mat/may/g # The method comment for {{ConfiguredFailoverProxyProvider#useLogicalURI}} says "Logical URI is not used for IP failover." While strictly true, I'm guessing you meant to put a different comment here. +1 once these are addressed. > Client failover proxy provider for IP failover based NN HA > -- > > Key: HDFS-6334 > URL: https://issues.apache.org/jira/browse/HDFS-6334 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Kihwal Lee >Assignee: Kihwal Lee > Attachments: HDFS-6334.patch, HDFS-6334.v2.patch > > > With RPCv9 and improvements in the SPNEGO auth handling, it is possible to > set up a pair of HA namenodes utilizing IP failover as client-request fencing > mechanism. > This jira will make it possible for HA to be configured without requiring use > of logical URI and provide a simple IP failover proxy provider. The change > will allow any old implementation of {{FailoverProxyProvider}} to continue to > work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6371) In HA setup, the standby NN should redirect WebHDFS write requests to the active NN
[ https://issues.apache.org/jira/browse/HDFS-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-6371: -- Component/s: namenode Issue Type: Improvement (was: Bug) > In HA setup, the standby NN should redirect WebHDFS write requests to the > active NN > --- > > Key: HDFS-6371 > URL: https://issues.apache.org/jira/browse/HDFS-6371 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, webhdfs >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > > The current WebHDFS implementation in namenode does not check its HA state -- > it does the same thing no matter it is active or standby. > Suppose a http client talk to the standby NN via WebHDFS. For the read > operations, there is no problem. For the write operations, if the operation > requires http redirect (e.g. creating a file), it will work since the standby > NN will also redirect the client to a DN. When the client connect to the DN, > the DN will fulfill the request with the active NN. However, for the write > operations not requiring http redirect (e.g. mkdir), the operation will fail > with StandbyException since it will be executed with the standby NN. > There are two solutions: > # The http client could catch StandbyException and then retries with the > other NN in this case. > # The standby NN redirects the request to the active NN. > The second solution seems better since the client does not need to know both > active NN and standby NN. > Note that WebHdfsFileSystem is already able to handle HA failover. The JIRA > is for other http clients. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6186) Pause deletion of blocks when the namenode starts up
[ https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996006#comment-13996006 ] Hadoop QA commented on HDFS-6186: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644511/HDFS-6186.003.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestCheckpoint {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6884//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6884//console This message is automatically generated. > Pause deletion of blocks when the namenode starts up > > > Key: HDFS-6186 > URL: https://issues.apache.org/jira/browse/HDFS-6186 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Suresh Srinivas >Assignee: Jing Zhao > Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch, > HDFS-6186.003.patch > > > HDFS namenode can delete blocks very quickly, given the deletion happens as a > parallel operation spread across many datanodes. One of the frequent > anxieties I see is that a lot of data can be deleted very quickly, when a > cluster is brought up, especially when one of the storage directories has > failed and namenode metadata was copied from another storage. Copying wrong > metadata would results in some of the newer files (if old metadata was > copied) being deleted along with their blocks. > HDFS-5986 now captures the number of pending deletion block on namenode webUI > and JMX. I propose pausing deletion of blocks for a configured period of time > (default 1 hour?) after namenode comes out of safemode. This will give enough > time for the administrator to notice large number of pending deletion blocks > and take corrective action. > Thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5682) Heterogeneous Storage phase 2 - APIs to expose Storage Types
[ https://issues.apache.org/jira/browse/HDFS-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996033#comment-13996033 ] Nick Dimiduk commented on HDFS-5682: Hi [~wuzesheng]. Can you describe your SLA requirements in detail? Maybe these needs can be addressed in other ways? > Heterogeneous Storage phase 2 - APIs to expose Storage Types > > > Key: HDFS-5682 > URL: https://issues.apache.org/jira/browse/HDFS-5682 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Phase 1 (HDFS-2832) added support to present the DataNode as a collection of > discrete storages of different types. > This Jira is to track phase 2 of the Heterogeneous Storage work which > involves exposing Storage Types to applications and adding Quota Management > support for administrators. > This phase will also include tools support for administrators/users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6372) Handle setXattr rpcIDs for OfflineEditsViewer
[ https://issues.apache.org/jira/browse/HDFS-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-6372. --- Resolution: Fixed Fix Version/s: HDFS XAttrs (HDFS-2006) Hadoop Flags: Reviewed > Handle setXattr rpcIDs for OfflineEditsViewer > - > > Key: HDFS-6372 > URL: https://issues.apache.org/jira/browse/HDFS-6372 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS XAttrs (HDFS-2006) > > Attachments: HDFS-6372.patch, editsStored > > > toXml and fromXml methods in SetXattrOp should store and read the rpcids. > This JIRA can also fix the failures of > TestOfflineEditsViewer > TestRetryCacheWithHA -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995978#comment-13995978 ] Yi Liu commented on HDFS-6283: -- Thanks Andrew. Sure, you can pick this one up:) > Write end user documentation for xattrs. > > > Key: HDFS-6283 > URL: https://issues.apache.org/jira/browse/HDFS-6283 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Reporter: Chris Nauroth >Assignee: Andrew Wang > Attachments: hdfs-6283-1.patch > > > Update the File System Shell documentation to cover the new getfattr and > setfattr commands. If warranted, consider adding a separate dedicated page > for fuller discussion of the xattrs model and how the feature works in more > detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5682) Heterogeneous Storage phase 2 - APIs to expose Storage Types
[ https://issues.apache.org/jira/browse/HDFS-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995299#comment-13995299 ] Arpit Agarwal commented on HDFS-5682: - Hi [~wuzesheng], this was deprioritized due to 2.4 stabilization. We will post a document describing the proposed API within the next couple of weeks, followed shortly by the detailed design doc. > Heterogeneous Storage phase 2 - APIs to expose Storage Types > > > Key: HDFS-5682 > URL: https://issues.apache.org/jira/browse/HDFS-5682 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Phase 1 (HDFS-2832) added support to present the DataNode as a collection of > discrete storages of different types. > This Jira is to track phase 2 of the Heterogeneous Storage work which > involves exposing Storage Types to applications and adding Quota Management > support for administrators. > This phase will also include tools support for administrators/users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks
[ https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995982#comment-13995982 ] Yi Liu commented on HDFS-6373: -- Thanks Andrew. Let's remove XAttrFeature from INodeSysmlink, so symlinks don't have xattrs of their own. In design doc we have {quote} Symlinks do not have XAttrs of their own. Operations that modify the XAttrs of a sysmlink instead modify the XAttr of the symlinkâs target {quote} > Remove support for extended attributes on symlinks > -- > > Key: HDFS-6373 > URL: https://issues.apache.org/jira/browse/HDFS-6373 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Andrew Wang >Assignee: Charles Lamb > > Looking in the Linux source code, we see the following: > http://lxr.linux.no/linux+v3.14.3/fs/xattr.c > {code} > 60/* > 61 * In the user.* namespace, only regular files and directories > can have > 62 * extended attributes. For sticky directories, only the owner and > 63 * privileged users can write attributes. > 64 */ > {code} > We should consider removing {{XAttrFeature}} from {{INodeSymlink}}. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Attachment: HDFS-6325.patch > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, > HDFS-6325_test.patch, appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
[ https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996031#comment-13996031 ] Hadoop QA commented on HDFS-6361: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12644540/HDFS-6361.002.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-nfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6887//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/6887//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-nfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6887//console This message is automatically generated. > TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id > - > > Key: HDFS-6361 > URL: https://issues.apache.org/jira/browse/HDFS-6361 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6361.001.patch, HDFS-6361.002.patch > > > The following error happens pretty often: > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting > Failing for the past 1 build (Since Unstable#61 ) > Took 0.1 sec. > add description > Error Message > For input string: "4294967294" > Stacktrace > java.lang.NumberFormatException: For input string: "4294967294" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:495) > at java.lang.Integer.valueOf(Integer.java:582) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188) > at org.apache.hadoop.nfs.nfs3.IdUserGroup.(IdUserGroup.java:60) > at > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71) > Standard Output > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.nfs.nfs3.IdUserGroup). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6344) Maximum limit on the size of an xattr
[ https://issues.apache.org/jira/browse/HDFS-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995968#comment-13995968 ] Yi Liu commented on HDFS-6344: -- Thanks Chris. Make sense, consistency is important. Let's address it in HDFS-6377 > Maximum limit on the size of an xattr > - > > Key: HDFS-6344 > URL: https://issues.apache.org/jira/browse/HDFS-6344 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Andrew Wang >Assignee: Yi Liu > Fix For: HDFS XAttrs (HDFS-2006) > > Attachments: HDFS-6344.patch > > > We should support limits on the maximum size of an xattr name/value. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-4437) Clients should not retain leases on moved files
[ https://issues.apache.org/jira/browse/HDFS-4437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HDFS-4437. Resolution: Duplicate Target Version/s: 2.1.0-beta, 3.0.0 (was: 3.0.0, 2.1.0-beta) > Clients should not retain leases on moved files > --- > > Key: HDFS-4437 > URL: https://issues.apache.org/jira/browse/HDFS-4437 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.0, 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > > Moving open files and/or parent directories of open files will rewrite the > leases to the new path. The client is not notified so future stream > operations will fail. However, as long as the client keeps its lease active > it will have a "lock" on the file in its new location. This is not good for > a daemon. > Leases should be released after the file is moved. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
[ https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-6361: Attachment: HDFS-6361.002.patch > TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id > - > > Key: HDFS-6361 > URL: https://issues.apache.org/jira/browse/HDFS-6361 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6361.001.patch, HDFS-6361.002.patch > > > The following error happens pretty often: > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting > Failing for the past 1 build (Since Unstable#61 ) > Took 0.1 sec. > add description > Error Message > For input string: "4294967294" > Stacktrace > java.lang.NumberFormatException: For input string: "4294967294" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:495) > at java.lang.Integer.valueOf(Integer.java:582) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188) > at org.apache.hadoop.nfs.nfs3.IdUserGroup.(IdUserGroup.java:60) > at > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71) > Standard Output > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.nfs.nfs3.IdUserGroup). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6313) WebHdfs may use the wrong NN when configured for multiple HA NNs
[ https://issues.apache.org/jira/browse/HDFS-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994350#comment-13994350 ] Hudson commented on HDFS-6313: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #560 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/560/]) HDFS-6313. WebHdfs may use the wrong NN when configured for multiple HA NNs. Contributed by Kihwal Lee. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593475) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFSForHA.java > WebHdfs may use the wrong NN when configured for multiple HA NNs > > > Key: HDFS-6313 > URL: https://issues.apache.org/jira/browse/HDFS-6313 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Kihwal Lee >Priority: Blocker > Fix For: 3.0.0, 2.4.1 > > Attachments: HDFS-6313.branch-2.4.patch, HDFS-6313.patch > > > WebHdfs resolveNNAddr will return a union of addresses for all HA configured > NNs. The client may access the wrong NN. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6345) DFS.listCacheDirectives() should allow filtering based on cache directive ID
[ https://issues.apache.org/jira/browse/HDFS-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6345: -- Target Version/s: 2.5.0 Status: Patch Available (was: In Progress) > DFS.listCacheDirectives() should allow filtering based on cache directive ID > > > Key: HDFS-6345 > URL: https://issues.apache.org/jira/browse/HDFS-6345 > Project: Hadoop HDFS > Issue Type: Bug > Components: caching >Affects Versions: 2.4.0 >Reporter: Lenni Kuff >Assignee: Andrew Wang > Attachments: hdfs-6345-1.patch > > > DFS.listCacheDirectives() should allow filtering based on cache directive ID. > Currently it throws an exception. > For example: > {code} > long directiveId = ; > CacheDirectiveInfo filter = new CacheDirectiveInfo.Builder() > .setId(directiveId) > .build(); > RemoteIterator itr = dfs.listCacheDirectives(filter); > {code} -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6372) Handle setXattr rpcIDs for OfflineEditsViewer
Uma Maheswara Rao G created HDFS-6372: - Summary: Handle setXattr rpcIDs for OfflineEditsViewer Key: HDFS-6372 URL: https://issues.apache.org/jira/browse/HDFS-6372 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G toXml and fromXml methods in SetXattrOp should store and read the rpcids. -- This message was sent by Atlassian JIRA (v6.2#6252)