[jira] [Updated] (HDFS-6346) Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call
[
https://issues.apache.org/jira/browse/HDFS-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-6346:
-
Attachment: HDFS-6346.1.patch
Thanks Uma for the review.
{quote}
Do you think, we need warn/log to the user that xattr attempting to remove does
not exist?
{quote}
make sense, let's add a log when removing nonexistent xattr.
{quote}
Pls remove empty line between
{quote}
OK
{quote}
When existing and updated xattrs equal then we need not even call
updateINodeXAttr. So code can be changed to
{code}
if (existingXAttrs.size() != newXAttrs.size()) {
XAttrStorage.updateINodeXAttrs(inode, newXAttrs, snapshotId);
return xAttr;
}
return null;
{code}
{quote}
Right, I will update it
> Optimize OP_SET_XATTRS by persisting single Xattr entry per
> setXattr/removeXattr api call
> -
>
> Key: HDFS-6346
> URL: https://issues.apache.org/jira/browse/HDFS-6346
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Uma Maheswara Rao G
>Assignee: Yi Liu
> Attachments: HDFS-6346.1.patch, HDFS-6346.2.patch, HDFS-6346.patch,
> editsStored
>
>
> When a new xattrs set on an Inode, it may add this with OP_SET_XATTRS and
> let's say [user.name1:value1]
> On a next call if we set another xattrs, then it may store along with older
> existing xattrs again. It may be like [user.name1:value1, user.name2:value2]
> So, on adding more xattrs on same Inode, that list may grow and we keep store
> the entries of already existing name, value fairs.
> Right now we defaulted the max Xattrs on an Inode to 32 and configured. If
> user modified it to much larger value and start setting xattrs, then edits
> loading may create issue like my above example.
> But I didn't refer any usecase of having large number of xattrs, this is just
> the assumption to consider a case. My biggest doubt is whether we will have
> such real usecases to have huge xattrs on a single INode.
> So, here is a thought on having OP_SET_XATTR for each setXAttr operation to
> be logged, When we replay them we need to consolidate. This is some initial
> thought we can think more if others also feel we need to consider this case
> to handle.
> Otherwise we endup storing Xattrs entries in editlog file as n(n+1)/2 where n
> is number xattrs for a file/dir. This may be issue only when we have large
> number configured max xattrs for inode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6381) Fix a typo in INodeReference.java
[
https://issues.apache.org/jira/browse/HDFS-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6381:
Resolution: Fixed
Fix Version/s: 2.5.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this to trunk and branch-2. Thanks for the fix [~decster]!
> Fix a typo in INodeReference.java
> -
>
> Key: HDFS-6381
> URL: https://issues.apache.org/jira/browse/HDFS-6381
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: Binglin Chang
>Assignee: Binglin Chang
>Priority: Trivial
> Fix For: 2.5.0
>
> Attachments: HDFS-6381.v1.patch
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
> {code}
> * For example,
> - * (1) Support we have /abc/foo, say the inode of foo is
> inode(id=1000,name=foo)
> + * (1) Suppose we have /abc/foo, say the inode of foo is
> inode(id=1000,name=foo)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.
[
https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997300#comment-13997300
]
Uma Maheswara Rao G commented on HDFS-6283:
---
{quote}
extended attributes are not interpreted by the system and are instead used by
applications to store additional information about an inode.
{quote}
I saw this line. This is saying Xattrs can not be interpreted by System. But
the xattrs with system/security name will be used only by system. So, we should
not generically say that Xattrs can not be interpreted by system? Please
correct me I interpreted this line wrongly :-)
Let me know if we need to change this, then I wil file a follow-up JIRA as it
is already committed.
> Write end user documentation for xattrs.
>
>
> Key: HDFS-6283
> URL: https://issues.apache.org/jira/browse/HDFS-6283
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
>Reporter: Chris Nauroth
>Assignee: Andrew Wang
> Fix For: HDFS XAttrs (HDFS-2006)
>
> Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch, hdfs-6283-3.patch
>
>
> Update the File System Shell documentation to cover the new getfattr and
> setfattr commands. If warranted, consider adding a separate dedicated page
> for fuller discussion of the xattrs model and how the feature works in more
> detail.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6293: - Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Release Note: Set "dfs.namenode.legacy-oiv-image.dir" to an appropriate directory to make standby name node or secondary name node save its file system state in the old fsimage format during checkpointing. This image can be used for offline analysis using the OfflineImageViewer. Use the "hdfs oiv_legacy" command to process the old fsimage format. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk and branch-2. Thanks for discussions, code and reviews! > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch, > HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch, > HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6394) HDFS encryption documentation
Alejandro Abdelnur created HDFS-6394: Summary: HDFS encryption documentation Key: HDFS-6394 URL: https://issues.apache.org/jira/browse/HDFS-6394 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Charles Lamb Documentation for HDFS encryption behavior and configuration -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2006) ability to support storing extended attributes per file
[
https://issues.apache.org/jira/browse/HDFS-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997264#comment-13997264
]
Uma Maheswara Rao G commented on HDFS-2006:
---
Thanks Nicholas, we will update the details in design doc about the size limit.
{code}
Also, I think we should have different limits for different XAttr namespaces
(user, trusted, security and system) since we may want to put a smaller limit
for the user visible namespaces. How does it sound?
{code}
This make sense to me. As in the initial thinking, we will have configuration
parameter for controlling the size. Now as you pointed, if at all Kernel wants
use it with a little larger value for some purpose later, we need to modify
the single parameter right now, from then it allows users also to write larger
values. Storing larger values in editlog is not good with any namespace, but it
will up to kernel implementation decision later.
I am +1 on this consideration.
Let me file a JIRA for it and discuss more about this point if others have any
concerns.
> ability to support storing extended attributes per file
> ---
>
> Key: HDFS-2006
> URL: https://issues.apache.org/jira/browse/HDFS-2006
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: HDFS XAttrs (HDFS-2006)
>Reporter: dhruba borthakur
>Assignee: Yi Liu
> Attachments: HDFS-XAttrs-Design-1.pdf, HDFS-XAttrs-Design-2.pdf,
> HDFS-XAttrs-Design-3.pdf, Test-Plan-for-Extended-Attributes-1.pdf,
> xattrs.1.patch, xattrs.patch
>
>
> It would be nice if HDFS provides a feature to store extended attributes for
> files, similar to the one described here:
> http://en.wikipedia.org/wiki/Extended_file_attributes.
> The challenge is that it has to be done in such a way that a site not using
> this feature does not waste precious memory resources in the namenode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks
[
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996856#comment-13996856
]
Andrew Wang commented on HDFS-6373:
---
That stacktrace is probably just from how symlink resolution works. The client
code catches the UnresolvedLinkException, attempts to resolve it, then retries
with the new path. Maybe we should squish the trace since this is expected
behavior.
> Remove support for extended attributes on symlinks
> --
>
> Key: HDFS-6373
> URL: https://issues.apache.org/jira/browse/HDFS-6373
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Andrew Wang
>Assignee: Charles Lamb
>
> Looking in the Linux source code, we see the following:
> http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
> {code}
> 60/*
> 61 * In the user.* namespace, only regular files and directories
> can have
> 62 * extended attributes. For sticky directories, only the owner and
> 63 * privileged users can write attributes.
> 64 */
> {code}
> We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996968#comment-13996968 ] Kihwal Lee commented on HDFS-6293: -- Thanks for the review, Suresh. I will commit this after precomit comes back. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch, > HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch, > HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2006) ability to support storing extended attributes per file
[ https://issues.apache.org/jira/browse/HDFS-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997288#comment-13997288 ] Yi Liu commented on HDFS-2006: -- Thanks Nicholas, I agree we have different limits for different XAttr namespaces. > ability to support storing extended attributes per file > --- > > Key: HDFS-2006 > URL: https://issues.apache.org/jira/browse/HDFS-2006 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: dhruba borthakur >Assignee: Yi Liu > Attachments: HDFS-XAttrs-Design-1.pdf, HDFS-XAttrs-Design-2.pdf, > HDFS-XAttrs-Design-3.pdf, Test-Plan-for-Extended-Attributes-1.pdf, > xattrs.1.patch, xattrs.patch > > > It would be nice if HDFS provides a feature to store extended attributes for > files, similar to the one described here: > http://en.wikipedia.org/wiki/Extended_file_attributes. > The challenge is that it has to be done in such a way that a site not using > this feature does not waste precious memory resources in the namenode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6366) FsImage loading failed with RemoveXattr op
[ https://issues.apache.org/jira/browse/HDFS-6366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994182#comment-13994182 ] Yi Liu commented on HDFS-6366: -- +1 for the patch. Thanks Uma. > FsImage loading failed with RemoveXattr op > -- > > Key: HDFS-6366 > URL: https://issues.apache.org/jira/browse/HDFS-6366 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS XAttrs (HDFS-2006) > > Attachments: HDFS-6366.patch > > > Should break after the removeXattr op loading. > Otherwise fsimage loading will fail: > java.io.IOException: Invalid operation read OP_REMOVE_XATTR > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:816) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:227) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:136) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:805) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:665) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:272) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:902) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:651) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:479) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:535) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:694) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:679) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1332) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:973) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:854) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:700) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:373) > at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:354) > at > org.apache.hadoop.hdfs.server.namenode.FSXAttrBaseTest.initCluster(FSXAttrBaseTest.java:400) > at > org.apache.hadoop.hdfs.server.namenode.FSXAttrBaseTest.restart(FSXAttrBaseTest.java:418) > at > org.apache.hadoop.hdfs.server.namenode.FSXAttrBaseTest.testCreateXAttr(FSXAttrBaseTest.java:121) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6367) Domain>#parse in EnumSetParam fails for parmater containing more than one enum.
Yi Liu created HDFS-6367: Summary: Domain>#parse in EnumSetParam fails for parmater containing more than one enum. Key: HDFS-6367 URL: https://issues.apache.org/jira/browse/HDFS-6367 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 3.0.0 Reporter: Yi Liu Fix For: 3.0.0 Attachments: HDFS-6367.patch Fails because additional "," java.lang.IllegalArgumentException: No enum const class org.apache.hadoop.fs.Options$Rename.,OVERWRITE at java.lang.Enum.valueOf(Enum.java:196) at org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85) at org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.(RenameOptionSetParam.java:45) at org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6376) Distcp data between two HA clusters requires another configuration
[ https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Marion updated HDFS-6376: -- Fix Version/s: 2.4.1 > Distcp data between two HA clusters requires another configuration > -- > > Key: HDFS-6376 > URL: https://issues.apache.org/jira/browse/HDFS-6376 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, federation, hdfs-client >Affects Versions: 2.3.0, 2.4.0 > Environment: CDH 5.0 (Apache 2.3.0 + some patches) >Reporter: Dave Marion > Fix For: 2.4.1 > > Attachments: HDFS-6376-2.patch, HDFS-6376-patch-1.patch > > > User has to create a third set of configuration files for distcp when > transferring data between two HA clusters. > Consider the scenario in [1]. You cannot put all of the required properties > in core-site.xml and hdfs-site.xml for the client to resolve the location of > both active namenodes. If you do, then the datanodes from cluster A may join > cluster B. I can not find a configuration option that tells the datanodes to > federate blocks for only one of the clusters in the configuration. > [1] > http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6383) Upgrade S3n s3.fs.buffer.dir to suppoer multi directories
[
https://issues.apache.org/jira/browse/HDFS-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997184#comment-13997184
]
Hadoop QA commented on HDFS-6383:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644622/HDFS-6383.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-common-project/hadoop-common:
org.apache.hadoop.http.TestHttpServer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6897//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6897//console
This message is automatically generated.
> Upgrade S3n s3.fs.buffer.dir to suppoer multi directories
> -
>
> Key: HDFS-6383
> URL: https://issues.apache.org/jira/browse/HDFS-6383
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.4.0
>Reporter: Ted Malaska
>Assignee: Ted Malaska
>Priority: Minor
> Attachments: HDFS-6383.patch
>
>
> s3.fs.buffer.dir defines the tmp folder where files will be written to before
> getting sent to S3. Right now this is limited to a single folder which
> causes to major issues.
> 1. You need a drive with enough space to store all the tmp files at once
> 2. You are limited to the IO speeds of a single drive
> This solution will resolve both and has been tested to increase the S3 write
> speed by 2.5x with 10 mappers on hs1.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-6283. --- Resolution: Fixed Fix Version/s: HDFS XAttrs (HDFS-2006) Committed to branch. > Write end user documentation for xattrs. > > > Key: HDFS-6283 > URL: https://issues.apache.org/jira/browse/HDFS-6283 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Reporter: Chris Nauroth >Assignee: Andrew Wang > Fix For: HDFS XAttrs (HDFS-2006) > > Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch, hdfs-6283-3.patch > > > Update the File System Shell documentation to cover the new getfattr and > setfattr commands. If warranted, consider adding a separate dedicated page > for fuller discussion of the xattrs model and how the feature works in more > detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2006) ability to support storing extended attributes per file
[ https://issues.apache.org/jira/browse/HDFS-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997202#comment-13997202 ] Tsz Wo Nicholas Sze commented on HDFS-2006: --- Thanks [~cnauroth] for showing me the pointer. I will look at the code in more details. Although large value is not yet supported in the current phase, the "XAttr with large value" section in the design doc should mention that if large value is supported, the large values should not be written to edit log. Otherwise, edit log will grow very fast. Also, I think we should have different limits for different XAttr namespaces (user, trusted, security and system) since we may want to put a smaller limit for the user visible namespaces. How does it sound? > ability to support storing extended attributes per file > --- > > Key: HDFS-2006 > URL: https://issues.apache.org/jira/browse/HDFS-2006 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: dhruba borthakur >Assignee: Yi Liu > Attachments: HDFS-XAttrs-Design-1.pdf, HDFS-XAttrs-Design-2.pdf, > HDFS-XAttrs-Design-3.pdf, Test-Plan-for-Extended-Attributes-1.pdf, > xattrs.1.patch, xattrs.patch > > > It would be nice if HDFS provides a feature to store extended attributes for > files, similar to the one described here: > http://en.wikipedia.org/wiki/Extended_file_attributes. > The challenge is that it has to be done in such a way that a site not using > this feature does not waste precious memory resources in the namenode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6293: - Attachment: HDFS-6293.trunk.patch The patch addresses review comments. I also added a simple check for OIV image generation in TestStandbyCheckpoints. Retention enforcement is tested in TestCheckpoint. This patch applies to trunk and branch-2. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.trunk.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6383) Upgrade S3n s3.fs.buffer.dir to suppoer multi directories
[ https://issues.apache.org/jira/browse/HDFS-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997211#comment-13997211 ] David S. Wang commented on HDFS-6383: - Thanks Ted for the patch. We should probably use the LocalDirAllocator like what s3a uses. That seems to be the proper way to do this in Hadoop. > Upgrade S3n s3.fs.buffer.dir to suppoer multi directories > - > > Key: HDFS-6383 > URL: https://issues.apache.org/jira/browse/HDFS-6383 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Ted Malaska >Assignee: Ted Malaska >Priority: Minor > Attachments: HDFS-6383.patch > > > s3.fs.buffer.dir defines the tmp folder where files will be written to before > getting sent to S3. Right now this is limited to a single folder which > causes to major issues. > 1. You need a drive with enough space to store all the tmp files at once > 2. You are limited to the IO speeds of a single drive > This solution will resolve both and has been tested to increase the S3 write > speed by 2.5x with 10 mappers on hs1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2006) ability to support storing extended attributes per file
[ https://issues.apache.org/jira/browse/HDFS-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996507#comment-13996507 ] Tsz Wo Nicholas Sze commented on HDFS-2006: --- The latest design looks good. For a XAttr with large value, writing it to editlog may be a problem. Do we plan to have a size limit on the size of XAttr values? > ability to support storing extended attributes per file > --- > > Key: HDFS-2006 > URL: https://issues.apache.org/jira/browse/HDFS-2006 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: dhruba borthakur >Assignee: Yi Liu > Attachments: HDFS-XAttrs-Design-1.pdf, HDFS-XAttrs-Design-2.pdf, > HDFS-XAttrs-Design-3.pdf, Test-Plan-for-Extended-Attributes-1.pdf, > xattrs.1.patch, xattrs.patch > > > It would be nice if HDFS provides a feature to store extended attributes for > files, similar to the one described here: > http://en.wikipedia.org/wiki/Extended_file_attributes. > The challenge is that it has to be done in such a way that a site not using > this feature does not waste precious memory resources in the namenode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6396) Remove support for ACL feature from INodeSymlink
Andrew Wang created HDFS-6396: - Summary: Remove support for ACL feature from INodeSymlink Key: HDFS-6396 URL: https://issues.apache.org/jira/browse/HDFS-6396 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Andrew Wang Assignee: Charles Lamb Priority: Minor Symlinks cannot have ACLs, but we still have support for the ACL feature in INodeSymlink because of class inheritance. Let's remove this support for code consistency. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6375) Listing extended attributes with the search permission
[
https://issues.apache.org/jira/browse/HDFS-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997120#comment-13997120
]
Charles Lamb commented on HDFS-6375:
Alternatively, we could just look at the permissions and only return the names
(or names + values) based on the permissions that the caller has.
> Listing extended attributes with the search permission
> --
>
> Key: HDFS-6375
> URL: https://issues.apache.org/jira/browse/HDFS-6375
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS XAttrs (HDFS-2006)
>Reporter: Andrew Wang
>Assignee: Charles Lamb
>
> From the attr(5) manpage:
> {noformat}
>Users with search access to a file or directory may retrieve a list of
>attribute names defined for that file or directory.
> {noformat}
> This is like doing {{getfattr}} without the {{-d}} flag, which we currently
> don't support.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6376) Distcp data between two HA clusters requires another configuration
[ https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Marion updated HDFS-6376: -- Component/s: hdfs-client datanode > Distcp data between two HA clusters requires another configuration > -- > > Key: HDFS-6376 > URL: https://issues.apache.org/jira/browse/HDFS-6376 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, federation, hdfs-client >Affects Versions: 2.3.0, 2.4.0 > Environment: CDH 5.0 (Apache 2.3.0 + some patches) >Reporter: Dave Marion > Fix For: 2.4.1 > > Attachments: HDFS-6376-2.patch, HDFS-6376-patch-1.patch > > > User has to create a third set of configuration files for distcp when > transferring data between two HA clusters. > Consider the scenario in [1]. You cannot put all of the required properties > in core-site.xml and hdfs-site.xml for the client to resolve the location of > both active namenodes. If you do, then the datanodes from cluster A may join > cluster B. I can not find a configuration option that tells the datanodes to > federate blocks for only one of the clusters in the configuration. > [1] > http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6381) Fix a typo in INodeReference.java
[
https://issues.apache.org/jira/browse/HDFS-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen He updated HDFS-6381:
--
Component/s: documentation
> Fix a typo in INodeReference.java
> -
>
> Key: HDFS-6381
> URL: https://issues.apache.org/jira/browse/HDFS-6381
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: Binglin Chang
>Assignee: Binglin Chang
>Priority: Trivial
> Attachments: HDFS-6381.v1.patch
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
> {code}
> * For example,
> - * (1) Support we have /abc/foo, say the inode of foo is
> inode(id=1000,name=foo)
> + * (1) Suppose we have /abc/foo, say the inode of foo is
> inode(id=1000,name=foo)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996873#comment-13996873 ] Kihwal Lee commented on HDFS-6293: -- I will fix all warnings except the two coming from FSImage due to the use of deprecated old image saving methods. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch, > HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6387) HDFS CLI admin tool for creating & deleting an encryption zone
Alejandro Abdelnur created HDFS-6387: Summary: HDFS CLI admin tool for creating & deleting an encryption zone Key: HDFS-6387 URL: https://issues.apache.org/jira/browse/HDFS-6387 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Charles Lamb CLI admin tool to create/delete an encryption zone in HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6377) Unify xattr name and value limits into a single limit
[
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997065#comment-13997065
]
Chris Nauroth commented on HDFS-6377:
-
Ah, thanks for the explanation of the {{cacheManager}} bit. :-)
> Unify xattr name and value limits into a single limit
> -
>
> Key: HDFS-6377
> URL: https://issues.apache.org/jira/browse/HDFS-6377
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS XAttrs (HDFS-2006)
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Fix For: HDFS XAttrs (HDFS-2006)
>
> Attachments: hdfs-6377-1.patch, hdfs-6377-2.patch
>
>
> Instead of having separate limits and config options for the size of an
> xattr's name and value, let's use a single limit.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6373) Remove support for extended attributes on symlinks
[
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-6373:
--
Resolution: Fixed
Fix Version/s: HDFS XAttrs (HDFS-2006)
Status: Resolved (was: Patch Available)
Committed to branch, thanks all.
> Remove support for extended attributes on symlinks
> --
>
> Key: HDFS-6373
> URL: https://issues.apache.org/jira/browse/HDFS-6373
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Andrew Wang
>Assignee: Charles Lamb
> Fix For: HDFS XAttrs (HDFS-2006)
>
> Attachments: HDFS-6373.1.patch, HDFS-6373.2.patch
>
>
> Looking in the Linux source code, we see the following:
> http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
> {code}
> 60/*
> 61 * In the user.* namespace, only regular files and directories
> can have
> 62 * extended attributes. For sticky directories, only the owner and
> 63 * privileged users can write attributes.
> 64 */
> {code}
> We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
[ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996841#comment-13996841 ] samira elhami commented on HDFS-2576: - This patch is really helpful. I know it is added to hadoop-2.1.0-Beta. I've install this version of hadoop but I don't know how to set the block placement. I've searched everywhere. please help me. Thanks > Namenode should have a favored nodes hint to enable clients to have control > over block placement. > - > > Key: HDFS-2576 > URL: https://issues.apache.org/jira/browse/HDFS-2576 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode >Reporter: Pritam Damania >Assignee: Devaraj Das > Fix For: 2.1.0-beta > > Attachments: hdfs-2576-1.txt, hdfs-2576-trunk-1.patch, > hdfs-2576-trunk-2.patch, hdfs-2576-trunk-7.1.patch, hdfs-2576-trunk-7.patch, > hdfs-2576-trunk-8.1.patch, hdfs-2576-trunk-8.2.patch, > hdfs-2576-trunk-8.3.patch, hdfs-2576-trunk-8.patch > > > Sometimes Clients like HBase are required to dynamically compute the > datanodes it wishes to place the blocks for a file for higher level of > locality. For this purpose there is a need of a way to give the Namenode a > hint in terms of a favoredNodes parameter about the locations where the > client wants to put each block. The proposed solution is a favored nodes > parameter in the addBlock() method and in the create() file method to enable > the clients to give the hints to the NameNode about the locations of each > replica of the block. Note that this would be just a hint and finally the > NameNode would look at disk usage, datanode load etc. and decide whether it > can respect the hints or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997142#comment-13997142 ] Kihwal Lee commented on HDFS-6293: -- The failed test is unrelated to this JIRA. It will be fixed in HDFS-6257. The two warnings (deprecation) are expected. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch, > HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch, > HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997053#comment-13997053 ] Andrew Wang commented on HDFS-6283: --- New patch attached, thanks for the reviews all. I think this addresses all the comments except as follows: - I didn't split out the common changes, seems like we could move this to a HADOOP subtask if people prefer, but it seems nicer to do the FsShell updates with this same patch as Chris requested. Plus, this way the typo is consistently fixed in all three places. - However, for wording changes, I'd prefer to punt that to another JIRA. I used the exactly text from the FsShell help, so we should update all three places (FsShell, ExtendedAttributes.apt.vm, FileSystemShell.apt.vm) if we want to change the wording anywhere. > Write end user documentation for xattrs. > > > Key: HDFS-6283 > URL: https://issues.apache.org/jira/browse/HDFS-6283 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Reporter: Chris Nauroth >Assignee: Andrew Wang > Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch > > > Update the File System Shell documentation to cover the new getfattr and > setfattr commands. If warranted, consider adding a separate dedicated page > for fuller discussion of the xattrs model and how the feature works in more > detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6395) Assorted improvements to xattr limit checking
Andrew Wang created HDFS-6395: - Summary: Assorted improvements to xattr limit checking Key: HDFS-6395 URL: https://issues.apache.org/jira/browse/HDFS-6395 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS XAttrs (HDFS-2006) Reporter: Andrew Wang It'd be nice to print messages during fsimage and editlog loading if we hit either the # of xattrs per inode or the xattr size limits. We should also consider making the # of xattrs limit only apply to the user namespace, or to each namespace separately, to prevent users from locking out access to other namespaces. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6377) Unify xattr name and value limits into a single limit
[ https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6377: -- Attachment: hdfs-6377-2.patch > Unify xattr name and value limits into a single limit > - > > Key: HDFS-6377 > URL: https://issues.apache.org/jira/browse/HDFS-6377 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-6377-1.patch, hdfs-6377-2.patch > > > Instead of having separate limits and config options for the size of an > xattr's name and value, let's use a single limit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6056) Clean up NFS config settings
[ https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997056#comment-13997056 ] Aaron T. Myers commented on HDFS-6056: -- Why not add proper config deprecations of the old config names to the new config names so that this won't be an incompatible change? That seems vastly preferable to me. > Clean up NFS config settings > > > Key: HDFS-6056 > URL: https://issues.apache.org/jira/browse/HDFS-6056 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Aaron T. Myers >Assignee: Brandon Li > Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, > HDFS-6056.003.patch, HDFS-6056.004.patch > > > As discussed on HDFS-6050, there's a few opportunities to improve the config > settings related to NFS. This JIRA is to implement those changes, which > include: moving hdfs-nfs related properties into hadoop-hdfs-nfs project, and > replacing 'nfs' with 'nfs' in the property names. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6293: - Status: Patch Available (was: In Progress) > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.trunk.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997051#comment-13997051
]
Hadoop QA commented on HDFS-6293:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644696/HDFS-6293.v2.trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1277 javac
compiler warnings (more than the trunk's current 1275 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6896//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6896//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6896//console
This message is automatically generated.
> Issues with OIV processing PB-based fsimages
>
>
> Key: HDFS-6293
> URL: https://issues.apache.org/jira/browse/HDFS-6293
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
> HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch,
> HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch,
> HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch,
> HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf.
> Due to the internal layout changes introduced by the protobuf-based fsimage,
> OIV consumes excessive amount of memory. We have tested with a fsimage with
> about 140M files/directories. The peak heap usage when processing this image
> in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
> the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
> heap (max new size was 1GB). It should be possible to process any image with
> the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.
> I also noticed that the secret manager section has no tokens while there were
> unexpired tokens in the original image (pre-2.4.0). I did not check whether
> they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6385) Show when block deletion will start after NameNode startup in WebUI
Jing Zhao created HDFS-6385: --- Summary: Show when block deletion will start after NameNode startup in WebUI Key: HDFS-6385 URL: https://issues.apache.org/jira/browse/HDFS-6385 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao HDFS-6186 provides functionality to delay block deletion for a period of time after NameNode startup. Currently we only show the number of pending block deletions in WebUI. We should also show when the block deletion will start in WebUI. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[
https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996582#comment-13996582
]
Hadoop QA commented on HDFS-6293:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644631/HDFS-6293.trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6892//console
This message is automatically generated.
> Issues with OIV processing PB-based fsimages
>
>
> Key: HDFS-6293
> URL: https://issues.apache.org/jira/browse/HDFS-6293
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch,
> HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.trunk.patch,
> HDFS-6293_sbn_ckpt_retention.patch,
> HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html
>
>
> There are issues with OIV when processing fsimages in protobuf.
> Due to the internal layout changes introduced by the protobuf-based fsimage,
> OIV consumes excessive amount of memory. We have tested with a fsimage with
> about 140M files/directories. The peak heap usage when processing this image
> in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting
> the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of
> heap (max new size was 1GB). It should be possible to process any image with
> the default heap size of 1.5GB.
> Another issue is the complete change of format/content in OIV's XML output.
> I also noticed that the secret manager section has no tokens while there were
> unexpired tokens in the original image (pre-2.4.0). I did not check whether
> they were also missing in the new pb fsimage.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6283: -- Attachment: hdfs-6283-2.patch > Write end user documentation for xattrs. > > > Key: HDFS-6283 > URL: https://issues.apache.org/jira/browse/HDFS-6283 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Reporter: Chris Nauroth >Assignee: Andrew Wang > Attachments: hdfs-6283-1.patch, hdfs-6283-2.patch > > > Update the File System Shell documentation to cover the new getfattr and > setfattr commands. If warranted, consider adding a separate dedicated page > for fuller discussion of the xattrs model and how the feature works in more > detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6334) Client failover proxy provider for IP failover based NN HA
[
https://issues.apache.org/jira/browse/HDFS-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-6334:
-
Resolution: Fixed
Fix Version/s: 2.5.0
3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the review, Aaron. I've committed this to trunk and branch-2.
> Client failover proxy provider for IP failover based NN HA
> --
>
> Key: HDFS-6334
> URL: https://issues.apache.org/jira/browse/HDFS-6334
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HDFS-6334.patch, HDFS-6334.v2.patch, HDFS-6334.v3.patch
>
>
> With RPCv9 and improvements in the SPNEGO auth handling, it is possible to
> set up a pair of HA namenodes utilizing IP failover as client-request fencing
> mechanism.
> This jira will make it possible for HA to be configured without requiring use
> of logical URI and provide a simple IP failover proxy provider. The change
> will allow any old implementation of {{FailoverProxyProvider}} to continue to
> work.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks
[
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996975#comment-13996975
]
Chris Nauroth commented on HDFS-6373:
-
{quote}
This makes sense to me, since symlink permissions are always ignored. Linux
actually doesn't even let you change symlink permissions, though it is allowed
by the Unix spec. Chris Nauroth, could you comment on whether this was
intentional, or just an oversight? It seems like we might want to remove
ACLs-on-symlinks, which is compatible since symlinks are still hard-disabled.
{quote}
It is impossible to set an ACL on a symlink. Any ACL operation run on a
symlink automatically dereferences the symlink and operates on its target, i.e.
by passing {{true}} to {{FSDirectory#getINodesInPath4Write}}. We wrote tests
asserting this behavior, so I don't think there is any way to store an
{{AclFeature}} on an {{INodeSymlink}}.
I quickly skimmed the patch, and it looks like you're going for some additional
prevention inside {{INodeSymlink}} to prevent mistakes. That seems like a good
idea, and we could do similar for ACLs.
> Remove support for extended attributes on symlinks
> --
>
> Key: HDFS-6373
> URL: https://issues.apache.org/jira/browse/HDFS-6373
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Andrew Wang
>Assignee: Charles Lamb
> Attachments: HDFS-6373.1.patch, HDFS-6373.2.patch
>
>
> Looking in the Linux source code, we see the following:
> http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
> {code}
> 60/*
> 61 * In the user.* namespace, only regular files and directories
> can have
> 62 * extended attributes. For sticky directories, only the owner and
> 63 * privileged users can write attributes.
> 64 */
> {code}
> We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6313) WebHdfs may use the wrong NN when configured for multiple HA NNs
[ https://issues.apache.org/jira/browse/HDFS-6313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993042#comment-13993042 ] Kihwal Lee commented on HDFS-6313: -- MiniDFSCluster or NN does not alter the config after its up. It was a easy way to get HA config set up for the test. > WebHdfs may use the wrong NN when configured for multiple HA NNs > > > Key: HDFS-6313 > URL: https://issues.apache.org/jira/browse/HDFS-6313 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Kihwal Lee >Priority: Blocker > Fix For: 3.0.0, 2.4.1 > > Attachments: HDFS-6313.branch-2.4.patch, HDFS-6313.patch > > > WebHdfs resolveNNAddr will return a union of addresses for all HA configured > NNs. The client may access the wrong NN. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks
[
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996912#comment-13996912
]
Charles Lamb commented on HDFS-6373:
Yeah, good idea. I've uploaded a new patch.
> Remove support for extended attributes on symlinks
> --
>
> Key: HDFS-6373
> URL: https://issues.apache.org/jira/browse/HDFS-6373
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Andrew Wang
>Assignee: Charles Lamb
> Attachments: HDFS-6373.1.patch, HDFS-6373.2.patch
>
>
> Looking in the Linux source code, we see the following:
> http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
> {code}
> 60/*
> 61 * In the user.* namespace, only regular files and directories
> can have
> 62 * extended attributes. For sticky directories, only the owner and
> 63 * privileged users can write attributes.
> 64 */
> {code}
> We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6370) Web UI fails to display in intranet under IE
[ https://issues.apache.org/jira/browse/HDFS-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6370: Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) I committed this to trunk and branch-2. Haohui, thank you for contributing the fix. > Web UI fails to display in intranet under IE > > > Key: HDFS-6370 > URL: https://issues.apache.org/jira/browse/HDFS-6370 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, journal-node, namenode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6370.000.patch > > > When IE renders the web UI of a cluster than runs in the intranet, it forces > the compatibility mode to be turned on which causes the UI fails to render > correctly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HDFS-6346) Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call
[ https://issues.apache.org/jira/browse/HDFS-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6346 started by Yi Liu. > Optimize OP_SET_XATTRS by persisting single Xattr entry per > setXattr/removeXattr api call > - > > Key: HDFS-6346 > URL: https://issues.apache.org/jira/browse/HDFS-6346 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Yi Liu > Attachments: HDFS-6346.patch, editsStored > > > When a new xattrs set on an Inode, it may add this with OP_SET_XATTRS and > let's say [user.name1:value1] > On a next call if we set another xattrs, then it may store along with older > existing xattrs again. It may be like [user.name1:value1, user.name2:value2] > So, on adding more xattrs on same Inode, that list may grow and we keep store > the entries of already existing name, value fairs. > Right now we defaulted the max Xattrs on an Inode to 32 and configured. If > user modified it to much larger value and start setting xattrs, then edits > loading may create issue like my above example. > But I didn't refer any usecase of having large number of xattrs, this is just > the assumption to consider a case. My biggest doubt is whether we will have > such real usecases to have huge xattrs on a single INode. > So, here is a thought on having OP_SET_XATTR for each setXAttr operation to > be logged, When we replay them we need to consolidate. This is some initial > thought we can think more if others also feel we need to consider this case > to handle. > Otherwise we endup storing Xattrs entries in editlog file as n(n+1)/2 where n > is number xattrs for a file/dir. This may be issue only when we have large > number configured max xattrs for inode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6373) Remove support for extended attributes on symlinks
[
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-6373:
---
Attachment: HDFS-6373.1.patch
Andrew, attached is a patch which overrides the appropriate xattr methods in
INodeWithAdditionalFields.
> Remove support for extended attributes on symlinks
> --
>
> Key: HDFS-6373
> URL: https://issues.apache.org/jira/browse/HDFS-6373
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Andrew Wang
>Assignee: Charles Lamb
> Attachments: HDFS-6373.1.patch
>
>
> Looking in the Linux source code, we see the following:
> http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
> {code}
> 60/*
> 61 * In the user.* namespace, only regular files and directories
> can have
> 62 * extended attributes. For sticky directories, only the owner and
> 63 * privileged users can write attributes.
> 64 */
> {code}
> We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996922#comment-13996922 ] Suresh Srinivas edited comment on HDFS-6293 at 5/13/14 9:36 PM: +1 for the patch. Thanks [~wheat9] and [~kihwal] for making these changes. was (Author: sureshms): +1 for the patch. Thanks [~hmai] and [~kihwal] for making these changes. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch, > HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch, > HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6395) Assorted improvements to xattr limit checking
[
https://issues.apache.org/jira/browse/HDFS-6395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996989#comment-13996989
]
Andrew Wang commented on HDFS-6395:
---
Let's also remove the extra import in FSNamesystem that Chris noticed from
HDFS-6377:
{code}
+import java.io.UnsupportedEncodingException;
{code}
> Assorted improvements to xattr limit checking
> -
>
> Key: HDFS-6395
> URL: https://issues.apache.org/jira/browse/HDFS-6395
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS XAttrs (HDFS-2006)
>Reporter: Andrew Wang
>
> It'd be nice to print messages during fsimage and editlog loading if we hit
> either the # of xattrs per inode or the xattr size limits.
> We should also consider making the # of xattrs limit only apply to the user
> namespace, or to each namespace separately, to prevent users from locking out
> access to other namespaces.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6373) Remove support for extended attributes on symlinks
[
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-6373:
---
Attachment: HDFS-6373.2.patch
> Remove support for extended attributes on symlinks
> --
>
> Key: HDFS-6373
> URL: https://issues.apache.org/jira/browse/HDFS-6373
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Andrew Wang
>Assignee: Charles Lamb
> Attachments: HDFS-6373.1.patch, HDFS-6373.2.patch
>
>
> Looking in the Linux source code, we see the following:
> http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
> {code}
> 60/*
> 61 * In the user.* namespace, only regular files and directories
> can have
> 62 * extended attributes. For sticky directories, only the owner and
> 63 * privileged users can write attributes.
> 64 */
> {code}
> We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6370) Web UI fails to display in intranet under IE
[
https://issues.apache.org/jira/browse/HDFS-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996809#comment-13996809
]
Hadoop QA commented on HDFS-6370:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644321/HDFS-6370.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6893//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6893//console
This message is automatically generated.
> Web UI fails to display in intranet under IE
>
>
> Key: HDFS-6370
> URL: https://issues.apache.org/jira/browse/HDFS-6370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, journal-node, namenode
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6370.000.patch
>
>
> When IE renders the web UI of a cluster than runs in the intranet, it forces
> the compatibility mode to be turned on which causes the UI fails to render
> correctly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996922#comment-13996922 ] Suresh Srinivas commented on HDFS-6293: --- +1 for the patch. Thanks [~hmai] and [~kihwal] for making these changes. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch, > HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, HDFS-6293.v2.branch-2.patch, > HDFS-6293.v2.trunk.patch, HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6378) NFS: when portmap/rpcbind is not available, NFS registration should timeout instead of hanging
[ https://issues.apache.org/jira/browse/HDFS-6378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6378: - Description: NFS gateway should > NFS: when portmap/rpcbind is not available, NFS registration should timeout > instead of hanging > --- > > Key: HDFS-6378 > URL: https://issues.apache.org/jira/browse/HDFS-6378 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Reporter: Brandon Li > > NFS gateway should -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995608#comment-13995608 ] Alejandro Abdelnur commented on HDFS-6134: -- Owen, I think the docs posted in HADOOP-10150 will address most if not all your questions/concerns: https://issues.apache.org/jira/secure/attachment/12640388/HDFSDataAtRestEncryptionAlternatives.pdf https://issues.apache.org/jira/secure/attachment/12640389/HDFSDataatRestEncryptionProposal.pdf https://issues.apache.org/jira/secure/attachment/12640390/HDFSDataatRestEncryptionAttackVectors.pdf > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks
[
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996905#comment-13996905
]
Andrew Wang commented on HDFS-6373:
---
This is basically what I had in mind, but how do you feel about throwing some
kind of RuntimeException, e.g. UnsupportedOperationException? It should prevent
anyone from trying this in the future.
> Remove support for extended attributes on symlinks
> --
>
> Key: HDFS-6373
> URL: https://issues.apache.org/jira/browse/HDFS-6373
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Andrew Wang
>Assignee: Charles Lamb
> Attachments: HDFS-6373.1.patch
>
>
> Looking in the Linux source code, we see the following:
> http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
> {code}
> 60/*
> 61 * In the user.* namespace, only regular files and directories
> can have
> 62 * extended attributes. For sticky directories, only the owner and
> 63 * privileged users can write attributes.
> 64 */
> {code}
> We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6345) DFS.listCacheDirectives() should allow filtering based on cache directive ID
[
https://issues.apache.org/jira/browse/HDFS-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996783#comment-13996783
]
Colin Patrick McCabe commented on HDFS-6345:
{code}
+// Querying for a single ID
+final Long id = filter.getId();
+if (id != null) {
+ if (!directivesById.containsKey(id)) {
+throw new InvalidRequestException("Did not find requested id " + id);
+ }
+ prevId = id - 1;
}
{code}
This is a bit tricky, so I think we should add a comment to the code explaining
why we're changing prevId.
{code}
+
ArrayList replies =
new ArrayList(NUM_PRE_ALLOCATED_ENTRIES);
int numReplies = 0;
@@ -710,6 +718,11 @@ public void removeDirective(long id, FSPermissionChecker
pc)
}
CacheDirective curDirective = cur.getValue();
CacheDirectiveInfo info = cur.getValue().toInfo();
+ if (id != null &&
+ !(info.getId().equals(id))) {
+// If we didn't find the requested ID, we're done
+break;
+ }
{code}
Maybe change the comment to "if we didn't find, or filtered out the requested
ID". Filtering by ID is only one kind of filtering... if an additional filter
is set which eliminates the requested item, nothing will show up.
> DFS.listCacheDirectives() should allow filtering based on cache directive ID
>
>
> Key: HDFS-6345
> URL: https://issues.apache.org/jira/browse/HDFS-6345
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: caching
>Affects Versions: 2.4.0
>Reporter: Lenni Kuff
>Assignee: Andrew Wang
> Attachments: hdfs-6345-1.patch
>
>
> DFS.listCacheDirectives() should allow filtering based on cache directive ID.
> Currently it throws an exception.
> For example:
> {code}
> long directiveId = ;
> CacheDirectiveInfo filter = new CacheDirectiveInfo.Builder()
> .setId(directiveId)
> .build();
> RemoteIterator itr = dfs.listCacheDirectives(filter);
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6283) Write end user documentation for xattrs.
[ https://issues.apache.org/jira/browse/HDFS-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996767#comment-13996767 ] Chris Nauroth commented on HDFS-6283: - Hi, Andrew. The new content looks great. Here are a few comments: # Would you please add a hyperlink to the new doc in hadoop-project/src/site/site.xml? # Would you please also update FileSystemShell.apt.vm to add getfattr and setfattr? # The "Go Back" link is broken. It looks like some existing Go Back links are broken too in the published documentation. Let's fix this one here, and I filed HADOOP-10602 to clean up the others. # Let's mention that setting a value for an xattr is optional. (Some might find it helpful to set a valueless xattr to use as a kind of tag.) > Write end user documentation for xattrs. > > > Key: HDFS-6283 > URL: https://issues.apache.org/jira/browse/HDFS-6283 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation >Reporter: Chris Nauroth >Assignee: Andrew Wang > Attachments: hdfs-6283-1.patch > > > Update the File System Shell documentation to cover the new getfattr and > setfattr commands. If warranted, consider adding a separate dedicated page > for fuller discussion of the xattrs model and how the feature works in more > detail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6056) Clean up NFS config settings
[ https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6056: - Attachment: HDFS-6056.004.patch > Clean up NFS config settings > > > Key: HDFS-6056 > URL: https://issues.apache.org/jira/browse/HDFS-6056 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Aaron T. Myers >Assignee: Brandon Li > Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, > HDFS-6056.003.patch, HDFS-6056.004.patch > > > As discussed on HDFS-6050, there's a few opportunities to improve the config > settings related to NFS. This JIRA is to implement those changes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6370) Web UI fails to display in intranet under IE
[ https://issues.apache.org/jira/browse/HDFS-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996834#comment-13996834 ] Chris Nauroth commented on HDFS-6370: - bq. -1 tests included. The patch doesn't appear to include any new or modified tests. This patch changes HTML only. There are no new automated tests. It has been tested manually by browsing with IE. I'm going to commit this. > Web UI fails to display in intranet under IE > > > Key: HDFS-6370 > URL: https://issues.apache.org/jira/browse/HDFS-6370 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, journal-node, namenode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6370.000.patch > > > When IE renders the web UI of a cluster than runs in the intranet, it forces > the compatibility mode to be turned on which causes the UI fails to render > correctly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996649#comment-13996649 ] Kihwal Lee commented on HDFS-6293: -- Ouch. The patch applies to trunk but it introduces a duplicate import in the test case. I will upload separate patches. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.trunk.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6384) Provide an alternative offline file system analysis tool
Kihwal Lee created HDFS-6384: Summary: Provide an alternative offline file system analysis tool Key: HDFS-6384 URL: https://issues.apache.org/jira/browse/HDFS-6384 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Kihwal Lee Priority: Critical Per discussion in HDFS-6293, we need an alternative way of analyzing file system offline. Some use cases can be covered by post-processing the new PB-based fsimage without consuming much resources, but others may need a way to efficiently generating fully constructed name space. Once we have a better solution, we should deprecate NN's ability to save its partial state in the old fsimage format. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6383) Upgrade S3n s3.fs.buffer.dir to suppoer multi directories
[ https://issues.apache.org/jira/browse/HDFS-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Malaska updated HDFS-6383: -- Attachment: HDFS-6383.patch This solves the problem. Thanks to: Joe Prosser, Dave Wang, Andrei Savu and Govind Kamat for testing > Upgrade S3n s3.fs.buffer.dir to suppoer multi directories > - > > Key: HDFS-6383 > URL: https://issues.apache.org/jira/browse/HDFS-6383 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ted Malaska >Priority: Minor > Attachments: HDFS-6383.patch > > > s3.fs.buffer.dir defines the tmp folder where files will be written to before > getting sent to S3. Right now this is limited to a single folder which > causes to major issues. > 1. You need a drive with enough space to store all the tmp files at once > 2. You are limited to the IO speeds of a single drive > This solution will resolve both and has been tested to increase the S3 write > speed by 2.5x with 10 mappers on hs1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6367) EnumSetParam$Domain#parse fails for parameter containing more than one enum.
[ https://issues.apache.org/jira/browse/HDFS-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-6367: -- Description: Fails because additional "," java.lang.IllegalArgumentException: No enum const class org.apache.hadoop.fs.Options$Rename.,OVERWRITE at java.lang.Enum.valueOf(Enum.java:196) at org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85) at org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.(RenameOptionSetParam.java:45) at org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355) was: Fails because additional "," java.lang.IllegalArgumentException: No enum const class org.apache.hadoop.fs.Options$Rename.,OVERWRITE at java.lang.Enum.valueOf(Enum.java:196) at org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85) at org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.(RenameOptionSetParam.java:45) at org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) > EnumSetParam$Domain#parse fails for parameter containing more than one enum. > > > Key: HDFS-6367 > URL: https://issues.apache.org/jira/browse/HDFS-6367 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Fix For: 3.0.0 > > Attachments: HDFS-6367.patch > > > Fails because additional "," > > java.lang.IllegalArgumentException: No enum const class > org.apache.hadoop.fs.Options$Rename.,OVERWRITE > at java.lang.Enum.valueOf(Enum.java:196) > at > org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85) > at > org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.(RenameOptionSetParam.java:45) > at > org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355) > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6355) Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner
[
https://issues.apache.org/jira/browse/HDFS-6355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-6355:
---
Resolution: Fixed
Fix Version/s: 2.5.0
Status: Resolved (was: Patch Available)
committed, thanks
> Fix divide-by-zero, improper use of wall-clock time in BlockPoolSliceScanner
>
>
> Key: HDFS-6355
> URL: https://issues.apache.org/jira/browse/HDFS-6355
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Fix For: 2.5.0
>
> Attachments: HDFS-6355.001.patch
>
>
> BlockPoolSliceScanner uses {{Time.now}} to calculate an interval. But this
> is incorrect, since if the wall-clock time changes, we will end up setting
> the scan periods to a shorter or longer time than we configured.
> There is also a case where we may divide by zero if we get unlucky, because
> we calculate an interval and divide by it, without checking whether the
> interval is 0 milliseconds. This would produce an {{ArithmeticException}}
> since we are using longs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6056) Clean up NFS config settings
[ https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6056: - Description: As discussed on HDFS-6050, there's a few opportunities to improve the config settings related to NFS. This JIRA is to implement those changes, which include: moving hdfs-nfs related properties into hadoop-hdfs-nfs project, and replacing 'nfs' with 'nfs' in the property names. (was: As discussed on HDFS-6050, there's a few opportunities to improve the config settings related to NFS. This JIRA is to implement those changes.) > Clean up NFS config settings > > > Key: HDFS-6056 > URL: https://issues.apache.org/jira/browse/HDFS-6056 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Aaron T. Myers >Assignee: Brandon Li > Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, > HDFS-6056.003.patch, HDFS-6056.004.patch > > > As discussed on HDFS-6050, there's a few opportunities to improve the config > settings related to NFS. This JIRA is to implement those changes, which > include: moving hdfs-nfs related properties into hadoop-hdfs-nfs project, and > replacing 'nfs' with 'nfs' in the property names. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6377) Unify xattr name and value limits into a single limit
[
https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995997#comment-13995997
]
Uma Maheswara Rao G commented on HDFS-6377:
---
Thanks a lot, Andrew for filing this JIRA and for the patch.
Adding to Liu's and Chris comments, I have following comments:
- . {code}
"XAttr is too big, maximum size = " + nnConf.xattrMaxSize
+ + ", but the size is = " + xAttr.getName().length());
{code}
Seems like now max size is combination of name and value size together, but
here you considered on name#length?
- . {code}
public static void init() throws Exception {
+conf = new Configuration();
{code}
Lets use HdfsCOnfiguration? [ but no harm in using either :-) ]
- . {code}
fs.setXAttr(path, name1, longValue);
+ fs.setXAttr(path, "user.a", longValue);
{code}
you can use name1 for it?
-
Do you think we need to have minimum configuration limit? Lets say user
configured size as 3, then this is always invalid size as Namespace itself
occupy this space? [ I am not insisting, just to discuss this point ]
Atleast we can warn on startup saying, you can not add xattrs with configured
limit? [ I think 7 is the minimum size for adding xattr with user name space
where name and value can have one char each at least]
-
Thanks for the other cleanup.
Yes, I agree with Chris, having fs-limits for consistency make sense to me.
> Unify xattr name and value limits into a single limit
> -
>
> Key: HDFS-6377
> URL: https://issues.apache.org/jira/browse/HDFS-6377
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS XAttrs (HDFS-2006)
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: hdfs-6377-1.patch
>
>
> Instead of having separate limits and config options for the size of an
> xattr's name and value, let's use a single limit.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HDFS-6380) TestBalancerWithNodeGroup is failing in trunk
[
https://issues.apache.org/jira/browse/HDFS-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G resolved HDFS-6380.
---
Resolution: Duplicate
> TestBalancerWithNodeGroup is failing in trunk
> -
>
> Key: HDFS-6380
> URL: https://issues.apache.org/jira/browse/HDFS-6380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Uma Maheswara Rao G
>
> Error Message
> expected:<1800> but was:<1814>
> Stacktrace
> {noformat}
> java.lang.AssertionError: expected:<1800> but was:<1814>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:253)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6186) Pause deletion of blocks when the namenode starts up
[
https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996688#comment-13996688
]
Suresh Srinivas commented on HDFS-6186:
---
+1 for the patch.
We should create another jira to print along with current pending deletion
count displayed on namenode webUI, something like:
{{Pending Deletion Blocks: (Block deletion will resume after xxx seconds)}}
> Pause deletion of blocks when the namenode starts up
>
>
> Key: HDFS-6186
> URL: https://issues.apache.org/jira/browse/HDFS-6186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Suresh Srinivas
>Assignee: Jing Zhao
> Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch,
> HDFS-6186.003.patch
>
>
> HDFS namenode can delete blocks very quickly, given the deletion happens as a
> parallel operation spread across many datanodes. One of the frequent
> anxieties I see is that a lot of data can be deleted very quickly, when a
> cluster is brought up, especially when one of the storage directories has
> failed and namenode metadata was copied from another storage. Copying wrong
> metadata would results in some of the newer files (if old metadata was
> copied) being deleted along with their blocks.
> HDFS-5986 now captures the number of pending deletion block on namenode webUI
> and JMX. I propose pausing deletion of blocks for a configured period of time
> (default 1 hour?) after namenode comes out of safemode. This will give enough
> time for the administrator to notice large number of pending deletion blocks
> and take corrective action.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6346) Optimize OP_SET_XATTRS by persisting single Xattr entry per setXattr/removeXattr api call
[ https://issues.apache.org/jira/browse/HDFS-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-6346: - Attachment: HDFS-6346.2.patch Uma, thanks for review and collecting the improvement data. That's great. The new patch includes update for your comment. > Optimize OP_SET_XATTRS by persisting single Xattr entry per > setXattr/removeXattr api call > - > > Key: HDFS-6346 > URL: https://issues.apache.org/jira/browse/HDFS-6346 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Uma Maheswara Rao G >Assignee: Yi Liu > Attachments: HDFS-6346.1.patch, HDFS-6346.2.patch, HDFS-6346.patch, > editsStored > > > When a new xattrs set on an Inode, it may add this with OP_SET_XATTRS and > let's say [user.name1:value1] > On a next call if we set another xattrs, then it may store along with older > existing xattrs again. It may be like [user.name1:value1, user.name2:value2] > So, on adding more xattrs on same Inode, that list may grow and we keep store > the entries of already existing name, value fairs. > Right now we defaulted the max Xattrs on an Inode to 32 and configured. If > user modified it to much larger value and start setting xattrs, then edits > loading may create issue like my above example. > But I didn't refer any usecase of having large number of xattrs, this is just > the assumption to consider a case. My biggest doubt is whether we will have > such real usecases to have huge xattrs on a single INode. > So, here is a thought on having OP_SET_XATTR for each setXAttr operation to > be logged, When we replay them we need to consolidate. This is some initial > thought we can think more if others also feel we need to consider this case > to handle. > Otherwise we endup storing Xattrs entries in editlog file as n(n+1)/2 where n > is number xattrs for a file/dir. This may be issue only when we have large > number configured max xattrs for inode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6369) RemoteBlockReader#available() should call FSInputChecker.available()
[
https://issues.apache.org/jira/browse/HDFS-6369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995294#comment-13995294
]
Colin Patrick McCabe commented on HDFS-6369:
The calling code in {{DFSInputStream}} already takes into account where the end
of the block is. See this code in {{DFSInputStream}}:
{code}
if (pos <= targetPos && targetPos <= blockEnd) {
//
// If this seek is to a positive position in the current
// block, and this piece of data might already be lying in
// the TCP buffer, then just eat up the intervening data.
//
int diff = (int)(targetPos - pos);
if (diff <= blockReader.available()) {
...
{code}
Notice that we don't call {{available}} if {{targetPos}} is beyond {{blockEnd}}
Also, {{RemoteBlockReader}} is deprecated. See the comment at the top:
{code}
/**
* \@deprecated this is an old implementation that is being left around
* in case any issues spring up with the new {\@link RemoteBlockReader2}
implementation.
* It will be removed in the next release.
*/
{code}
We should really fix {{RemoteBlockReader2}} to support SOCKS sockets (those are
the only ones without channels, as I recall) and remove {{RemoteBlockReader}}
as the comment promised we'd do.
> RemoteBlockReader#available() should call FSInputChecker.available()
>
>
> Key: HDFS-6369
> URL: https://issues.apache.org/jira/browse/HDFS-6369
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Chen He
>Priority: Trivial
> Attachments: HDFS-6369.patch
>
>
> Currently DFSClient.TCP_WINDOW_SIZE is directly returned.
> However, FSInputChecker.available(), in the superclass, may return value
> lower than the constant.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6230) Expose upgrade status through NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mit Desai updated HDFS-6230: Attachment: HDFS-6230.patch Thanks for looking at the patch [~wheat9]. Posting the updated patch. > Expose upgrade status through NameNode web UI > - > > Key: HDFS-6230 > URL: https://issues.apache.org/jira/browse/HDFS-6230 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Arpit Agarwal >Assignee: Mit Desai > Attachments: HDFS-6230-NoUpgradesInProgress.png, > HDFS-6230-UpgradeInProgress.jpg, HDFS-6230.patch, HDFS-6230.patch > > > The NameNode web UI does not show upgrade information anymore. Hadoop 2.0 > also does not have the _hadoop dfsadmin -upgradeProgress_ command to check > the upgrade status. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6326) WebHdfs ACL compatibility is broken
[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-6326:
Attachment: HDFS-6326.5.patch
Daryn, that looks like a good suggestion to me. Here is patch v5 incorporating
those ideas. Mostly, this is just folding your code right into the v4 patch
with some additional comments. Note however that I kept the
{{FsAclPermission}} class as {{Private}} inside the HDFS package, not Common.
I'd prefer not to publish this into the public API footprint.
I was hesitant to add a boolean member to track true/false for ACL presence
because of the risk of increasing memory footprint. However, I confirmed with
jmap -histo that it makes no difference after your suggested change:
{code}
375: 1 32 org.apache.hadoop.fs.permission.FsPermission
382: 1 32
org.apache.hadoop.hdfs.protocol.FsAclPermission
{code}
Memory utilization of {{FsPermission}} and {{FsAclPermission}} are very much
dominated by Java object overhead/padding, so adding the extra boolean didn't
make a difference. Haohui and I have had some offline discussion about
optimizations here, but that's not directly relevant to this issue.
[~daryn], can you please let me know if you're still +1 for v5? Thanks!
> WebHdfs ACL compatibility is broken
> ---
>
> Key: HDFS-6326
> URL: https://issues.apache.org/jira/browse/HDFS-6326
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Daryn Sharp
>Assignee: Chris Nauroth
>Priority: Blocker
> Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch,
> HDFS-6326.4.patch, HDFS-6326.5.patch, aclfsperm.example
>
>
> 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN
> throws an {{IllegalArgumentException}} exception.
> {code}
> hadoop fs -ls webhdfs://nn/
> Found 21 items
> ls: Invalid value for webhdfs parameter "op": No enum constant
> org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
> [... 20 more times...]
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6367) Domain>#parse in EnumSetParam fails for parmater containing more than one enum.
[ https://issues.apache.org/jira/browse/HDFS-6367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-6367: - Attachment: HDFS-6367.patch The "," should not be contained. > Domain>#parse in EnumSetParam fails for parmater containing > more than one enum. > - > > Key: HDFS-6367 > URL: https://issues.apache.org/jira/browse/HDFS-6367 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0 >Reporter: Yi Liu > Fix For: 3.0.0 > > Attachments: HDFS-6367.patch > > > Fails because additional "," > > java.lang.IllegalArgumentException: No enum const class > org.apache.hadoop.fs.Options$Rename.,OVERWRITE > at java.lang.Enum.valueOf(Enum.java:196) > at > org.apache.hadoop.hdfs.web.resources.EnumSetParam$Domain.parse(EnumSetParam.java:85) > at > org.apache.hadoop.hdfs.web.resources.RenameOptionSetParam.(RenameOptionSetParam.java:45) > at > org.apache.hadoop.hdfs.web.resources.TestParam.testRenameOptionSetParam(TestParam.java:355) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6370) Web UI fails to display in intranet under IE
[ https://issues.apache.org/jira/browse/HDFS-6370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6370: Component/s: namenode journal-node datanode Target Version/s: 3.0.0, 2.5.0 Affects Version/s: 3.0.0 2.4.0 Hadoop Flags: Reviewed > Web UI fails to display in intranet under IE > > > Key: HDFS-6370 > URL: https://issues.apache.org/jira/browse/HDFS-6370 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, journal-node, namenode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6370.000.patch > > > When IE renders the web UI of a cluster than runs in the intranet, it forces > the compatibility mode to be turned on which causes the UI fails to render > correctly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6056) Clean up NFS config settings
[ https://issues.apache.org/jira/browse/HDFS-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996698#comment-13996698 ] Brandon Li commented on HDFS-6056: -- [~jingzhao], thanks for the review. I've update the jira description and marked it as incompatible change. I've uploaded a new patch to address your comments. > Clean up NFS config settings > > > Key: HDFS-6056 > URL: https://issues.apache.org/jira/browse/HDFS-6056 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.3.0 >Reporter: Aaron T. Myers >Assignee: Brandon Li > Attachments: HDFS-6056.001.patch, HDFS-6056.002.patch, > HDFS-6056.003.patch, HDFS-6056.004.patch > > > As discussed on HDFS-6050, there's a few opportunities to improve the config > settings related to NFS. This JIRA is to implement those changes, which > include: moving hdfs-nfs related properties into hadoop-hdfs-nfs project, and > replacing 'nfs' with 'nfs' in the property names. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6361) TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id
[ https://issues.apache.org/jira/browse/HDFS-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-6361: Attachment: HDFS-6361.003.patch > TestIdUserGroup.testUserUpdateSetting failed due to out of range nfsnobody Id > - > > Key: HDFS-6361 > URL: https://issues.apache.org/jira/browse/HDFS-6361 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6361.001.patch, HDFS-6361.002.patch, > HDFS-6361.003.patch > > > The following error happens pretty often: > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting > Failing for the past 1 build (Since Unstable#61 ) > Took 0.1 sec. > add description > Error Message > For input string: "4294967294" > Stacktrace > java.lang.NumberFormatException: For input string: "4294967294" > at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > at java.lang.Integer.parseInt(Integer.java:495) > at java.lang.Integer.valueOf(Integer.java:582) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMapInternal(IdUserGroup.java:137) > at > org.apache.hadoop.nfs.nfs3.IdUserGroup.updateMaps(IdUserGroup.java:188) > at org.apache.hadoop.nfs.nfs3.IdUserGroup.(IdUserGroup.java:60) > at > org.apache.hadoop.nfs.nfs3.TestIdUserGroup.testUserUpdateSetting(TestIdUserGroup.java:71) > Standard Output > log4j:WARN No appenders could be found for logger > (org.apache.hadoop.nfs.nfs3.IdUserGroup). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6340) DN can't finalize upgarde
[ https://issues.apache.org/jira/browse/HDFS-6340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13992964#comment-13992964 ] Arpit Agarwal commented on HDFS-6340: - It looks like the previous build aborted. I kicked off a new one. > DN can't finalize upgarde > - > > Key: HDFS-6340 > URL: https://issues.apache.org/jira/browse/HDFS-6340 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.4.0 >Reporter: Rahul Singhal >Priority: Blocker > Fix For: 3.0.0, 2.4.1 > > Attachments: HDFS-6340-branch-2.4.0.patch, HDFS-6340.02.patch, > HDFS-6340.patch > > > I upgraded a (NN) HA cluster from 2.2.0 to 2.4.0. After I issued the > '-finalizeUpgarde' command, NN was able to finalize the upgrade but DN > couldn't (I waited for the next block report). > I think I have found the problem to be due to HDFS-5153. I will attach a > proposed fix. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6357) SetXattr should persist rpcIDs for handling retrycache with Namenode restart and HA
[ https://issues.apache.org/jira/browse/HDFS-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994293#comment-13994293 ] Uma Maheswara Rao G commented on HDFS-6357: --- Thanks a lot, for the reviews Yi and Andrew! I have just committed this to branch! > SetXattr should persist rpcIDs for handling retrycache with Namenode restart > and HA > --- > > Key: HDFS-6357 > URL: https://issues.apache.org/jira/browse/HDFS-6357 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: HDFS XAttrs (HDFS-2006) > > Attachments: HDFS-6357.patch, HDFS-6357.patch, HDFS-6357.patch > > > SetXattr should persist rpcIDs for handling restart Namenode and HA scenarios -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996649#comment-13996649 ] Kihwal Lee edited comment on HDFS-6293 at 5/13/14 5:19 PM: --- Oops. The patch applies to trunk but it introduces a duplicate import in the test case. I will upload separate patches. was (Author: kihwal): Ouch. The patch applies to trunk but it introduces a duplicate import in the test case. I will upload separate patches. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, HDFS-6293.branch-2.patch, > HDFS-6293.trunk.patch, HDFS-6293.trunk.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5682) Heterogeneous Storage phase 2 - APIs to expose Storage Types
[ https://issues.apache.org/jira/browse/HDFS-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996072#comment-13996072 ] Zesheng Wu commented on HDFS-5682: -- Thanks [~ndimiduk], here are some details of our SLA requirements: The background is that we have a structured data storage system which is very similar with amazon DynamoDB and is built on HBase. This system is intended to serve multi-tenants, in order to ensure the SLA, such as 99% read/write latency, we need to place the data on SSD. One straightforward way is to set up a hdfs cluster with all SSD data disks, but this is cost too much. So we need to reduce the cost. Another way is to set up a hdfs cluster with both SSD and HDD data disks, we can put 1 replica on SSD, and the other 2 on HDD. This is really a typical scenario of hdfs heterogeneous storage. > Heterogeneous Storage phase 2 - APIs to expose Storage Types > > > Key: HDFS-5682 > URL: https://issues.apache.org/jira/browse/HDFS-5682 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Phase 1 (HDFS-2832) added support to present the DataNode as a collection of > discrete storages of different types. > This Jira is to track phase 2 of the Heterogeneous Storage work which > involves exposing Storage Types to applications and adding Quota Management > support for administrators. > This phase will also include tools support for administrators/users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6250) TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
[
https://issues.apache.org/jira/browse/HDFS-6250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996207#comment-13996207
]
Junping Du commented on HDFS-6250:
--
Hi [~decster], patch LGTM. Would you run 50 iterations and update us your
results here? Thx!
> TestBalancerWithNodeGroup.testBalancerWithRackLocality fails
>
>
> Key: HDFS-6250
> URL: https://issues.apache.org/jira/browse/HDFS-6250
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Kihwal Lee
>Assignee: Chen He
> Attachments: HDFS-6250-v2.patch, HDFS-6250-v3.patch, HDFS-6250.patch,
> test_log.txt
>
>
> It was seen in https://builds.apache.org/job/PreCommit-HDFS-Build/6669/
> {panel}
> java.lang.AssertionError: expected:<1800> but was:<1810>
> at org.junit.Assert.fail(Assert.java:93)
> at org.junit.Assert.failNotEquals(Assert.java:647)
> at org.junit.Assert.assertEquals(Assert.java:128)
> at org.junit.Assert.assertEquals(Assert.java:147)
> at org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
> .testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:253)
> {panel}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6326) WebHdfs ACL compatibility is broken
[
https://issues.apache.org/jira/browse/HDFS-6326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995821#comment-13995821
]
Chris Nauroth commented on HDFS-6326:
-
I've repeated my compatibility testing on patch v4 and confirmed that it still
works.
> WebHdfs ACL compatibility is broken
> ---
>
> Key: HDFS-6326
> URL: https://issues.apache.org/jira/browse/HDFS-6326
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Daryn Sharp
>Assignee: Chris Nauroth
>Priority: Blocker
> Attachments: HDFS-6326.1.patch, HDFS-6326.2.patch, HDFS-6326.3.patch,
> HDFS-6326.4.patch
>
>
> 2.4 ACL support is completely incompatible with <2.4 webhdfs servers. The NN
> throws an {{IllegalArgumentException}} exception.
> {code}
> hadoop fs -ls webhdfs://nn/
> Found 21 items
> ls: Invalid value for webhdfs parameter "op": No enum constant
> org.apache.hadoop.hdfs.web.resources.GetOpParam.Op.GETACLSTATUS
> [... 20 more times...]
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-6293: Assignee: Kihwal Lee (was: Haohui Mai) > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6371) In HA setup, the standby NN should redirect WebHDFS write requests to the active NN
[ https://issues.apache.org/jira/browse/HDFS-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996394#comment-13996394 ] Daryn Sharp commented on HDFS-6371: --- Agreed on client side retries. With IP-based failover, a standby NN that attempts to redirect to the other NN will create a redirect loop to itself until the IP failover to the active occurs. > In HA setup, the standby NN should redirect WebHDFS write requests to the > active NN > --- > > Key: HDFS-6371 > URL: https://issues.apache.org/jira/browse/HDFS-6371 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode, webhdfs >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > > The current WebHDFS implementation in namenode does not check its HA state -- > it does the same thing no matter it is active or standby. > Suppose a http client talk to the standby NN via WebHDFS. For the read > operations, there is no problem. For the write operations, if the operation > requires http redirect (e.g. creating a file), it will work since the standby > NN will also redirect the client to a DN. When the client connect to the DN, > the DN will fulfill the request with the active NN. However, for the write > operations not requiring http redirect (e.g. mkdir), the operation will fail > with StandbyException since it will be executed with the standby NN. > There are two solutions: > # The http client could catch StandbyException and then retries with the > other NN in this case. > # The standby NN redirects the request to the active NN. > The second solution seems better since the client does not need to know both > active NN and standby NN. > Note that WebHdfsFileSystem is already able to handle HA failover. The JIRA > is for other http clients. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6381) Fix a typo in INodeReference.java
[
https://issues.apache.org/jira/browse/HDFS-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996213#comment-13996213
]
Junping Du commented on HDFS-6381:
--
Kick off Jenkins test manually.
> Fix a typo in INodeReference.java
> -
>
> Key: HDFS-6381
> URL: https://issues.apache.org/jira/browse/HDFS-6381
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Binglin Chang
>Assignee: Binglin Chang
>Priority: Trivial
> Attachments: HDFS-6381.v1.patch
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeReference.java
> {code}
> * For example,
> - * (1) Support we have /abc/foo, say the inode of foo is
> inode(id=1000,name=foo)
> + * (1) Suppose we have /abc/foo, say the inode of foo is
> inode(id=1000,name=foo)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996442#comment-13996442 ] Kihwal Lee commented on HDFS-6293: -- Here is recap. * oiv_legacy for debugging and inspecting old fsimages. As long as NN is capable of reading and upgrading from it, we should keep it. * NN's ability to save in old fsimage format: This should go away once we have a more sustainable solution. Postprocessing ([~wheat9] 's proposal) will cover many use cases. We may need something else. This is to be determined. * Keep adding features to the old fsimage format: Please don't! * Changing the layout of the new PB-based fsimage back to the old way: Heck no! We will deprecate NN's ability to save in the old fsimage format, once we agree on what the long term solution is. We should create a separate JIRA and continue discussion. As [~sureshms] suggested, I will prepare the patch for trunk as well with [~ajisakaa]'s comment incorporated. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Haohui Mai >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6383) Upgrade S3n s3.fs.buffer.dir to suppoer multi directories
[ https://issues.apache.org/jira/browse/HDFS-6383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Malaska updated HDFS-6383: -- Status: Patch Available (was: Open) Patch is ready > Upgrade S3n s3.fs.buffer.dir to suppoer multi directories > - > > Key: HDFS-6383 > URL: https://issues.apache.org/jira/browse/HDFS-6383 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ted Malaska >Priority: Minor > Attachments: HDFS-6383.patch > > > s3.fs.buffer.dir defines the tmp folder where files will be written to before > getting sent to S3. Right now this is limited to a single folder which > causes to major issues. > 1. You need a drive with enough space to store all the tmp files at once > 2. You are limited to the IO speeds of a single drive > This solution will resolve both and has been tested to increase the S3 write > speed by 2.5x with 10 mappers on hs1. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6373) Remove support for extended attributes on symlinks
[
https://issues.apache.org/jira/browse/HDFS-6373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996425#comment-13996425
]
Charles Lamb commented on HDFS-6373:
While investigating this, I noticed this in the TestNameNodeXAttr output. I
suspect it's not intended, but I'll investigate further:
2014-05-13 10:04:33,584 INFO hdfs.StateChange
(FSNamesystem.java:completeFile(3028)) - DIR* completeFile: /symdir2/target is
closed by DFSClient_NONMAPREDUCE_239063403_1
2014-05-13 10:04:33,590 INFO FSNamesystem.audit
(FSNamesystem.java:logAuditMessage(7980)) - allowed=true ugi=cwl
(auth:SIMPLE) ip=/127.0.0.1 cmd=createSymlink src=/symdir1/link
dst=/symdir2/target perm=cwl:supergroup:rwxrwxrwx proto=rpc
2014-05-13 10:04:33,601 INFO FSNamesystem.audit
(FSNamesystem.java:logAuditMessage(7980)) - allowed=true ugi=cwl
(auth:SIMPLE) ip=/127.0.0.1 cmd=setXAttrsrc=/symdir2/target
dst=nullperm=cwl:supergroup:rw-r--r-- proto=rpc
2014-05-13 10:04:33,605 INFO FSNamesystem.audit
(FSNamesystem.java:logAuditMessage(7980)) - allowed=true ugi=cwl
(auth:SIMPLE) ip=/127.0.0.1 cmd=setXAttrsrc=/symdir2/target
dst=nullperm=cwl:supergroup:rw-r--r-- proto=rpc
2014-05-13 10:04:33,608 INFO ipc.Server (Server.java:run(2120)) - IPC Server
handler 9 on 38622, call
org.apache.hadoop.hdfs.protocol.ClientProtocol.getXAttrs from 127.0.0.1:41722
Call#18 Retry#0
org.apache.hadoop.hdfs.protocol.UnresolvedPathException: /symdir2/target
at
org.apache.hadoop.hdfs.server.namenode.INodesInPath.resolve(INodesInPath.java:203)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getLastINodeInPath(FSDirectory.java:3135)
at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.getXAttrs(FSDirectory.java:2952)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getXAttrs(FSNamesystem.java:7858)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getXAttrs(NameNodeRpcServer.java:1396)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getXAttrs(ClientNamenodeProtocolServerSideTranslatorPB.java:1294)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:605)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:932)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2099)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2095)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1608)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2093)
> Remove support for extended attributes on symlinks
> --
>
> Key: HDFS-6373
> URL: https://issues.apache.org/jira/browse/HDFS-6373
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Andrew Wang
>Assignee: Charles Lamb
>
> Looking in the Linux source code, we see the following:
> http://lxr.linux.no/linux+v3.14.3/fs/xattr.c
> {code}
> 60/*
> 61 * In the user.* namespace, only regular files and directories
> can have
> 62 * extended attributes. For sticky directories, only the owner and
> 63 * privileged users can write attributes.
> 64 */
> {code}
> We should consider removing {{XAttrFeature}} from {{INodeSymlink}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6186) Pause deletion of blocks when the namenode starts up
[
https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996194#comment-13996194
]
Hadoop QA commented on HDFS-6186:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644511/HDFS-6186.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6890//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6890//console
This message is automatically generated.
> Pause deletion of blocks when the namenode starts up
>
>
> Key: HDFS-6186
> URL: https://issues.apache.org/jira/browse/HDFS-6186
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Suresh Srinivas
>Assignee: Jing Zhao
> Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch,
> HDFS-6186.003.patch
>
>
> HDFS namenode can delete blocks very quickly, given the deletion happens as a
> parallel operation spread across many datanodes. One of the frequent
> anxieties I see is that a lot of data can be deleted very quickly, when a
> cluster is brought up, especially when one of the storage directories has
> failed and namenode metadata was copied from another storage. Copying wrong
> metadata would results in some of the newer files (if old metadata was
> copied) being deleted along with their blocks.
> HDFS-5986 now captures the number of pending deletion block on namenode webUI
> and JMX. I propose pausing deletion of blocks for a configured period of time
> (default 1 hour?) after namenode comes out of safemode. This will give enough
> time for the administrator to notice large number of pending deletion blocks
> and take corrective action.
> Thoughts?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6345) DFS.listCacheDirectives() should allow filtering based on cache directive ID
[
https://issues.apache.org/jira/browse/HDFS-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996149#comment-13996149
]
Hadoop QA commented on HDFS-6345:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644544/hdfs-6345-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDistributedFileSystem
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6888//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6888//console
This message is automatically generated.
> DFS.listCacheDirectives() should allow filtering based on cache directive ID
>
>
> Key: HDFS-6345
> URL: https://issues.apache.org/jira/browse/HDFS-6345
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: caching
>Affects Versions: 2.4.0
>Reporter: Lenni Kuff
>Assignee: Andrew Wang
> Attachments: hdfs-6345-1.patch
>
>
> DFS.listCacheDirectives() should allow filtering based on cache directive ID.
> Currently it throws an exception.
> For example:
> {code}
> long directiveId = ;
> CacheDirectiveInfo filter = new CacheDirectiveInfo.Builder()
> .setId(directiveId)
> .build();
> RemoteIterator itr = dfs.listCacheDirectives(filter);
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6377) Unify xattr name and value limits into a single limit
[ https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996396#comment-13996396 ] Charles Lamb commented on HDFS-6377: In general, LGTM. One nit (piling on Uma's comment): + "XAttr is too big, maximum size = " + nnConf.xattrMaxSize + + ", but the size is = " + xAttr.getName().length()); Could we change the wording to "The XAttr is too big. The maximum size of the name + value is , but the total size is Unify xattr name and value limits into a single limit > - > > Key: HDFS-6377 > URL: https://issues.apache.org/jira/browse/HDFS-6377 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: hdfs-6377-1.patch > > > Instead of having separate limits and config options for the size of an > xattr's name and value, let's use a single limit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995837#comment-13995837
]
Hadoop QA commented on HDFS-6325:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644461/HDFS-6325.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
org.apache.hadoop.hdfs.TestFileAppend4
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6879//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6879//console
This message is automatically generated.
> Append should fail if the last block has insufficient number of replicas
>
>
> Key: HDFS-6325
> URL: https://issues.apache.org/jira/browse/HDFS-6325
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.2.0
>Reporter: Konstantin Shvachko
>Assignee: Keith Pak
> Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch,
> appendTest.patch
>
>
> Currently append() succeeds on a file with the last block that has no
> replicas. But the subsequent updatePipeline() fails as there are no replicas
> with the exception "Unable to retrieve blocks locations for last block". This
> leaves the file unclosed, and others can not do anything with it until its
> lease expires.
> The solution is to check replicas of the last block on the NameNode and fail
> during append() rather than during updatePipeline().
> How many replicas should be present before NN allows to append? I see two
> options:
> # min-replication: allow append if the last block is minimally replicated (1
> by default)
> # full-replication: allow append if the last block is fully replicated (3 by
> default)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995444#comment-13995444 ] Kihwal Lee commented on HDFS-6293: -- bq. can you please mark the class that are brought back deprecated? I am not sure deprecating it now is a good idea. There are two apsects in this. * Ability to read the old fsimage format in OIV. This has always been the case for OIV. Now it will be the combination of oiv and oiv_legacy that will continue this. As long as NN can read/upgrade old images, there is no reason to hurry and remove this capability from oiv. * Support for the use case built around the old oiv. Users won't care much about the image format as long as they can obtain what they want, using reasonable and comparable amount of resources as before. If we only think about this, we can deprecate OIV and NN's ability to save the old format, once we have a long term alternative solution. > Issues with OIV processing PB-based fsimages > > > Key: HDFS-6293 > URL: https://issues.apache.org/jira/browse/HDFS-6293 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Kihwal Lee >Assignee: Haohui Mai >Priority: Blocker > Attachments: HDFS-6293.000.patch, HDFS-6293.001.patch, > HDFS-6293.002-save-deprecated-fsimage.patch, > HDFS-6293_sbn_ckpt_retention.patch, > HDFS-6293_sbn_ckpt_retention_oiv_legacy.patch, Heap Histogram.html > > > There are issues with OIV when processing fsimages in protobuf. > Due to the internal layout changes introduced by the protobuf-based fsimage, > OIV consumes excessive amount of memory. We have tested with a fsimage with > about 140M files/directories. The peak heap usage when processing this image > in pre-protobuf (i.e. pre-2.4.0) format was about 350MB. After converting > the image to the protobuf format on 2.4.0, OIV would OOM even with 80GB of > heap (max new size was 1GB). It should be possible to process any image with > the default heap size of 1.5GB. > Another issue is the complete change of format/content in OIV's XML output. > I also noticed that the secret manager section has no tokens while there were > unexpired tokens in the original image (pre-2.4.0). I did not check whether > they were also missing in the new pb fsimage. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6328) Clean up dead code in FSDirectory
[ https://issues.apache.org/jira/browse/HDFS-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995409#comment-13995409 ] Hudson commented on HDFS-6328: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5603 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5603/]) HDFS-6328. Clean up dead code in FSDirectory. Contributed by Haohui Mai. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1593755) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java > Clean up dead code in FSDirectory > - > > Key: HDFS-6328 > URL: https://issues.apache.org/jira/browse/HDFS-6328 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6328.000.patch, HDFS-6328.001.patch, > HDFS-6328.002.patch, HDFS-6328.003.patch, HDFS-6328.004.patch > > > This jira proposes: > # Cleaning up dead code in FSDirectory. > # Simplify the control flows that IntelliJ flags as warnings. > # Move functions related to resolving paths into one place. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6382) HDFS File/Directory TTL
Zesheng Wu created HDFS-6382: Summary: HDFS File/Directory TTL Key: HDFS-6382 URL: https://issues.apache.org/jira/browse/HDFS-6382 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client, namenode Affects Versions: 2.4.0 Reporter: Zesheng Wu In production environment, we always have scenario like this, we want to backup files on hdfs for some time and then hope to delete theses files automatically. For example, we keep only 1 day's logs on local disk due to limited disk space, but we need to keep about 1 month's logs in order to debug program bugs, so we keep all the logs on hdfs and delete logs which are older than 1 month. This is a typical scenario of HDFS TTL. So here we propose that hdfs can support TTL. Following are some details of this proposal: 1. HDFS can support TTL on a specified file or directory 2. If a TTL is set on a file, the file will be deleted automatically after the TTL is expired 3. If a TTL is set on a directory, the child files and directories will be deleted automatically after the TTL is expired 4. The child file/directory's TTL configuration should override its parent directory's 5. A global configuration is needed to configure that whether the deleted files/directories should go to the trash or not 6. A global configuration is needed to configure that whether a directory with TTL should be deleted when it is emptied by TTL mechanism or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6230) Expose upgrade status through NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-6230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995340#comment-13995340 ] Haohui Mai commented on HDFS-6230: -- +1. I'll commit it shortly. > Expose upgrade status through NameNode web UI > - > > Key: HDFS-6230 > URL: https://issues.apache.org/jira/browse/HDFS-6230 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: Arpit Agarwal >Assignee: Mit Desai > Attachments: HDFS-6230-NoUpgradesInProgress.png, > HDFS-6230-UpgradeInProgress.jpg, HDFS-6230.patch, HDFS-6230.patch > > > The NameNode web UI does not show upgrade information anymore. Hadoop 2.0 > also does not have the _hadoop dfsadmin -upgradeProgress_ command to check > the upgrade status. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6305) WebHdfs response decoding may throw RuntimeExceptions
[ https://issues.apache.org/jira/browse/HDFS-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995737#comment-13995737 ] Jonathan Eagles commented on HDFS-6305: --- Kicked jenkins. there are 3 or 4 builds ahead of this one. It may be late tonight to see the results. Jon > WebHdfs response decoding may throw RuntimeExceptions > - > > Key: HDFS-6305 > URL: https://issues.apache.org/jira/browse/HDFS-6305 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Critical > Attachments: HDFS-6305.patch > > > WebHdfs does not guard against exceptions while decoding the response > payload. The json parser will throw RunTime exceptions on malformed > responses. The json decoding routines do not validate the expected fields > are present which may cause NPEs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6379) HTTPFS - Implement ACLs support
Alejandro Abdelnur created HDFS-6379: Summary: HTTPFS - Implement ACLs support Key: HDFS-6379 URL: https://issues.apache.org/jira/browse/HDFS-6379 Project: Hadoop HDFS Issue Type: Bug Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.4.0 HDFS-4685 added ACLs support to WebHDFS but missed adding them to HttpFS. This JIRA is for such. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6345) DFS.listCacheDirectives() should allow filtering based on cache directive ID
[
https://issues.apache.org/jira/browse/HDFS-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-6345:
--
Attachment: hdfs-6345-1.patch
Patch attached. I did manual testing with a branch-2 NN and the new client,
verified that it still filters correctly, and that we get the same exception
thrown on error.
> DFS.listCacheDirectives() should allow filtering based on cache directive ID
>
>
> Key: HDFS-6345
> URL: https://issues.apache.org/jira/browse/HDFS-6345
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: caching
>Affects Versions: 2.4.0
>Reporter: Lenni Kuff
>Assignee: Andrew Wang
> Attachments: hdfs-6345-1.patch
>
>
> DFS.listCacheDirectives() should allow filtering based on cache directive ID.
> Currently it throws an exception.
> For example:
> {code}
> long directiveId = ;
> CacheDirectiveInfo filter = new CacheDirectiveInfo.Builder()
> .setId(directiveId)
> .build();
> RemoteIterator itr = dfs.listCacheDirectives(filter);
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996245#comment-13996245
]
Steve Loughran commented on HDFS-6134:
--
Some side comments
# The ASF don't want to be distributing cryptography libraries; the design has
to address that distribution model.
# HADOOP-9565 proposes a {{BlobStore}} extending FileSystem to let users know
that the FS doesn't have the semantics of a real filesystem -for blobstore
security we'd need the same marker, so apps using a wrapped blobstore will know
that some operations are non-atomic and potentially O(files)
> Transparent data at rest encryption
> ---
>
> Key: HDFS-6134
> URL: https://issues.apache.org/jira/browse/HDFS-6134
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.3.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFSDataAtRestEncryption.pdf
>
>
> Because of privacy and security regulations, for many industries, sensitive
> data at rest must be in encrypted form. For example: the healthÂcare industry
> (HIPAA regulations), the card payment industry (PCI DSS regulations) or the
> US government (FISMA regulations).
> This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
> be used transparently by any application accessing HDFS via Hadoop Filesystem
> Java API, Hadoop libhdfs C library, or WebHDFS REST API.
> The resulting implementation should be able to be used in compliance with
> different regulation requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6374) setXAttr should require the user to be the owner of the file or directory
[
https://issues.apache.org/jira/browse/HDFS-6374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995988#comment-13995988
]
Yi Liu commented on HDFS-6374:
--
setXAttr/removeXAttr are only for owner, and getXAttr are *not* only for Owner
and by "r" permission.
> setXAttr should require the user to be the owner of the file or directory
> -
>
> Key: HDFS-6374
> URL: https://issues.apache.org/jira/browse/HDFS-6374
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Affects Versions: HDFS XAttrs (HDFS-2006)
>Reporter: Andrew Wang
>Assignee: Charles Lamb
>
> From the attr(5) manpage:
> {noformat}
>For this reason, extended user attributes are only allowed for regular
>files and directories, and access to extended user attributes is
>restricted to the owner and to users with appropriate capabilities for
>directories with the sticky bit set (see the chmod(1) manual page for
>an explanation of Sticky Directories).
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6382) HDFS File/Directory TTL
[ https://issues.apache.org/jira/browse/HDFS-6382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zesheng Wu updated HDFS-6382: - Description: In production environment, we always have scenario like this, we want to backup files on hdfs for some time and then hope to delete these files automatically. For example, we keep only 1 day's logs on local disk due to limited disk space, but we need to keep about 1 month's logs in order to debug program bugs, so we keep all the logs on hdfs and delete logs which are older than 1 month. This is a typical scenario of HDFS TTL. So here we propose that hdfs can support TTL. Following are some details of this proposal: 1. HDFS can support TTL on a specified file or directory 2. If a TTL is set on a file, the file will be deleted automatically after the TTL is expired 3. If a TTL is set on a directory, the child files and directories will be deleted automatically after the TTL is expired 4. The child file/directory's TTL configuration should override its parent directory's 5. A global configuration is needed to configure that whether the deleted files/directories should go to the trash or not 6. A global configuration is needed to configure that whether a directory with TTL should be deleted when it is emptied by TTL mechanism or not. was: In production environment, we always have scenario like this, we want to backup files on hdfs for some time and then hope to delete theses files automatically. For example, we keep only 1 day's logs on local disk due to limited disk space, but we need to keep about 1 month's logs in order to debug program bugs, so we keep all the logs on hdfs and delete logs which are older than 1 month. This is a typical scenario of HDFS TTL. So here we propose that hdfs can support TTL. Following are some details of this proposal: 1. HDFS can support TTL on a specified file or directory 2. If a TTL is set on a file, the file will be deleted automatically after the TTL is expired 3. If a TTL is set on a directory, the child files and directories will be deleted automatically after the TTL is expired 4. The child file/directory's TTL configuration should override its parent directory's 5. A global configuration is needed to configure that whether the deleted files/directories should go to the trash or not 6. A global configuration is needed to configure that whether a directory with TTL should be deleted when it is emptied by TTL mechanism or not. > HDFS File/Directory TTL > --- > > Key: HDFS-6382 > URL: https://issues.apache.org/jira/browse/HDFS-6382 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, namenode >Affects Versions: 2.4.0 >Reporter: Zesheng Wu > > In production environment, we always have scenario like this, we want to > backup files on hdfs for some time and then hope to delete these files > automatically. For example, we keep only 1 day's logs on local disk due to > limited disk space, but we need to keep about 1 month's logs in order to > debug program bugs, so we keep all the logs on hdfs and delete logs which are > older than 1 month. This is a typical scenario of HDFS TTL. So here we > propose that hdfs can support TTL. > Following are some details of this proposal: > 1. HDFS can support TTL on a specified file or directory > 2. If a TTL is set on a file, the file will be deleted automatically after > the TTL is expired > 3. If a TTL is set on a directory, the child files and directories will be > deleted automatically after the TTL is expired > 4. The child file/directory's TTL configuration should override its parent > directory's > 5. A global configuration is needed to configure that whether the deleted > files/directories should go to the trash or not > 6. A global configuration is needed to configure that whether a directory > with TTL should be deleted when it is emptied by TTL mechanism or not. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6377) Unify xattr name and value limits into a single limit
[ https://issues.apache.org/jira/browse/HDFS-6377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-6377: - Assignee: Andrew Wang > Unify xattr name and value limits into a single limit > - > > Key: HDFS-6377 > URL: https://issues.apache.org/jira/browse/HDFS-6377 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS XAttrs (HDFS-2006) >Reporter: Andrew Wang >Assignee: Andrew Wang > > Instead of having separate limits and config options for the size of an > xattr's name and value, let's use a single limit. -- This message was sent by Atlassian JIRA (v6.2#6252)
