[jira] [Updated] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-6776:
Attachment: HDFS-6776.010.patch
distcp from insecure cluster (source) to secure cluster (destination) doesn't
work via webhdfs
--
Key: HDFS-6776
URL: https://issues.apache.org/jira/browse/HDFS-6776
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.3.0, 2.5.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch, HDFS-6776.008.patch,
HDFS-6776.009.patch, HDFS-6776.010.patch, dummy-token-proxy.js
Issuing distcp command at the secure cluster side, trying to copy stuff from
insecure cluster to secure cluster, and see the following problem:
{code}
hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://insure-cluster:port/tmp
hdfs://sure-cluster:8020/tmp/tmptgt
14/07/30 20:06:19 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null,
sourcePaths=[webhdfs://insecure-cluster:port/tmp],
targetPath=hdfs://secure-cluster:8020/tmp/tmptgt, targetPathExists=true}
14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
secure-clister:8032
14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
'ssl.client.truststore.location' has not been set, no TrustStore will be
loaded
14/07/30 20:06:20 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Failed to get the token for hadoopuser,
user=hadoopuser
14/07/30 20:06:20 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Failed to get the token for hadoopuser,
user=hadoopuser
14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
[jira] [Commented] (HDFS-6776) distcp from insecure cluster (source) to secure cluster (destination) doesn't work via webhdfs
[
https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113387#comment-14113387
]
Yongjun Zhang commented on HDFS-6776:
-
Hi [~atm], thanks a lot for the review and comments. I just uploaded rev 010 to
address them.
I usually test against real clusters before I upload new rev, but my clusters
are down now. I will test out tomorrow.
Thanks.
distcp from insecure cluster (source) to secure cluster (destination) doesn't
work via webhdfs
--
Key: HDFS-6776
URL: https://issues.apache.org/jira/browse/HDFS-6776
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.3.0, 2.5.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-6776.001.patch, HDFS-6776.002.patch,
HDFS-6776.003.patch, HDFS-6776.004.patch, HDFS-6776.004.patch,
HDFS-6776.005.patch, HDFS-6776.006.NullToken.patch,
HDFS-6776.006.NullToken.patch, HDFS-6776.007.patch, HDFS-6776.008.patch,
HDFS-6776.009.patch, HDFS-6776.010.patch, dummy-token-proxy.js
Issuing distcp command at the secure cluster side, trying to copy stuff from
insecure cluster to secure cluster, and see the following problem:
{code}
hadoopuser@yjc5u-1 ~]$ hadoop distcp webhdfs://insure-cluster:port/tmp
hdfs://sure-cluster:8020/tmp/tmptgt
14/07/30 20:06:19 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, maxMaps=20, sslConfigurationFile='null',
copyStrategy='uniformsize', sourceFileListing=null,
sourcePaths=[webhdfs://insecure-cluster:port/tmp],
targetPath=hdfs://secure-cluster:8020/tmp/tmptgt, targetPathExists=true}
14/07/30 20:06:19 INFO client.RMProxy: Connecting to ResourceManager at
secure-clister:8032
14/07/30 20:06:20 WARN ssl.FileBasedKeyStoresFactory: The property
'ssl.client.truststore.location' has not been set, no TrustStore will be
loaded
14/07/30 20:06:20 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Failed to get the token for hadoopuser,
user=hadoopuser
14/07/30 20:06:20 WARN security.UserGroupInformation:
PriviledgedActionException as:hadoopu...@xyz.com (auth:KERBEROS)
cause:java.io.IOException: Failed to get the token for hadoopuser,
user=hadoopuser
14/07/30 20:06:20 ERROR tools.DistCp: Exception encountered
java.io.IOException: Failed to get the token for hadoopuser, user=hadoopuser
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:365)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:84)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:618)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:584)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:466)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:462)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1132)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:218)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:403)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:424)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:640)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:565)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:438)
at
[jira] [Commented] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113415#comment-14113415 ] Karthik Kambatla commented on HDFS-6914: Do we really want to target this to 2.5.1? If not, I would suggest targetting 2.6. Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.1 Reporter: Hao Chen Labels: hdfs Fix For: 2.5.1 Attachments: HDFS-6914.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6831) Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
[
https://issues.apache.org/jira/browse/HDFS-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-6831:
-
Attachment: HDFS-6831.0.patch
Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
---
Key: HDFS-6831
URL: https://issues.apache.org/jira/browse/HDFS-6831
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: HDFS-6831.0.patch
There is an inconsistency between the console outputs of 'hdfs dfsadmin'
command and 'hdfs dfsadmin -help' command.
{code}
[root@trunk ~]# hdfs dfsadmin
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot snapshotDir]
[-disallowSnapshot snapshotDir]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-rollingUpgrade [query|prepare|finalize]]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota quota dirname...dirname]
[-clrQuota dirname...dirname]
[-setSpaceQuota quota dirname...dirname]
[-clrSpaceQuota dirname...dirname]
[-setBalancerBandwidth bandwidth in bytes per second]
[-fetchImage local directory]
[-shutdownDatanode datanode_host:ipc_port [upgrade]]
[-getDatanodeInfo datanode_host:ipc_port]
[-help [cmd]]
{code}
{code}
[root@trunk ~]# hdfs dfsadmin -help
hadoop dfsadmin performs DFS administrative commands.
The full syntax is:
hadoop dfsadmin
[-report [-live] [-dead] [-decommissioning]]
[-safemode enter | leave | get | wait]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-setQuota quota dirname...dirname]
[-clrQuota dirname...dirname]
[-setSpaceQuota quota dirname...dirname]
[-clrSpaceQuota dirname...dirname]
[-finalizeUpgrade]
[-rollingUpgrade [query|prepare|finalize]]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh host:ipc_port key [arg1..argn]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanodehost:port blockpoolId [force]]
[-setBalancerBandwidth bandwidth]
[-fetchImage local directory]
[-allowSnapshot snapshotDir]
[-disallowSnapshot snapshotDir]
[-shutdownDatanode datanode_host:ipc_port [upgrade]]
[-getDatanodeInfo datanode_host:ipc_port
[-help [cmd]
{code}
These two outputs should be the same.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6634) inotify in HDFS
[
https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113469#comment-14113469
]
Hadoop QA commented on HDFS-6634:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664819/HDFS-6634.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal:
org.apache.hadoop.security.TestRefreshUserMappings
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7803//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7803//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7803//console
This message is automatically generated.
inotify in HDFS
---
Key: HDFS-6634
URL: https://issues.apache.org/jira/browse/HDFS-6634
Project: Hadoop HDFS
Issue Type: New Feature
Components: hdfs-client, namenode, qjm
Reporter: James Thomas
Assignee: James Thomas
Attachments: HDFS-6634.2.patch, HDFS-6634.3.patch, HDFS-6634.4.patch,
HDFS-6634.5.patch, HDFS-6634.6.patch, HDFS-6634.7.patch, HDFS-6634.patch,
inotify-design.2.pdf, inotify-design.3.pdf, inotify-design.4.pdf,
inotify-design.pdf, inotify-intro.2.pdf, inotify-intro.pdf
Design a mechanism for applications like search engines to access the HDFS
edit stream.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6831) Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
[
https://issues.apache.org/jira/browse/HDFS-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113475#comment-14113475
]
Xiaoyu Yao commented on HDFS-6831:
--
Attach a patch that fixed the issue with a commonUsageSummary string for both
commands and misc minor fixes.
* add the missing -metasave filename
* add the Note: Administrative commands can only be run with superuser
permission.\n to summary of PrintHelp() as Jing suggested.
* change java DfsAdmin to hdfs DfsAdmin in PrintUsage()
Not sure why quota and rolling upgrade commands have different for display
format with two + instead of - and extra spaces. I just keep it as is and
guess maybe it is to differentiate as new features.
\t[ + SetQuotaCommand.USAGE + ]\n +
Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
---
Key: HDFS-6831
URL: https://issues.apache.org/jira/browse/HDFS-6831
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: HDFS-6831.0.patch
There is an inconsistency between the console outputs of 'hdfs dfsadmin'
command and 'hdfs dfsadmin -help' command.
{code}
[root@trunk ~]# hdfs dfsadmin
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot snapshotDir]
[-disallowSnapshot snapshotDir]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-rollingUpgrade [query|prepare|finalize]]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota quota dirname...dirname]
[-clrQuota dirname...dirname]
[-setSpaceQuota quota dirname...dirname]
[-clrSpaceQuota dirname...dirname]
[-setBalancerBandwidth bandwidth in bytes per second]
[-fetchImage local directory]
[-shutdownDatanode datanode_host:ipc_port [upgrade]]
[-getDatanodeInfo datanode_host:ipc_port]
[-help [cmd]]
{code}
{code}
[root@trunk ~]# hdfs dfsadmin -help
hadoop dfsadmin performs DFS administrative commands.
The full syntax is:
hadoop dfsadmin
[-report [-live] [-dead] [-decommissioning]]
[-safemode enter | leave | get | wait]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-setQuota quota dirname...dirname]
[-clrQuota dirname...dirname]
[-setSpaceQuota quota dirname...dirname]
[-clrSpaceQuota dirname...dirname]
[-finalizeUpgrade]
[-rollingUpgrade [query|prepare|finalize]]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh host:ipc_port key [arg1..argn]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanodehost:port blockpoolId [force]]
[-setBalancerBandwidth bandwidth]
[-fetchImage local directory]
[-allowSnapshot snapshotDir]
[-disallowSnapshot snapshotDir]
[-shutdownDatanode datanode_host:ipc_port [upgrade]]
[-getDatanodeInfo datanode_host:ipc_port
[-help [cmd]
{code}
These two outputs should be the same.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6831) Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
[
https://issues.apache.org/jira/browse/HDFS-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-6831:
-
Status: Patch Available (was: Open)
Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
---
Key: HDFS-6831
URL: https://issues.apache.org/jira/browse/HDFS-6831
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: HDFS-6831.0.patch
There is an inconsistency between the console outputs of 'hdfs dfsadmin'
command and 'hdfs dfsadmin -help' command.
{code}
[root@trunk ~]# hdfs dfsadmin
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot snapshotDir]
[-disallowSnapshot snapshotDir]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-rollingUpgrade [query|prepare|finalize]]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota quota dirname...dirname]
[-clrQuota dirname...dirname]
[-setSpaceQuota quota dirname...dirname]
[-clrSpaceQuota dirname...dirname]
[-setBalancerBandwidth bandwidth in bytes per second]
[-fetchImage local directory]
[-shutdownDatanode datanode_host:ipc_port [upgrade]]
[-getDatanodeInfo datanode_host:ipc_port]
[-help [cmd]]
{code}
{code}
[root@trunk ~]# hdfs dfsadmin -help
hadoop dfsadmin performs DFS administrative commands.
The full syntax is:
hadoop dfsadmin
[-report [-live] [-dead] [-decommissioning]]
[-safemode enter | leave | get | wait]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-setQuota quota dirname...dirname]
[-clrQuota dirname...dirname]
[-setSpaceQuota quota dirname...dirname]
[-clrSpaceQuota dirname...dirname]
[-finalizeUpgrade]
[-rollingUpgrade [query|prepare|finalize]]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh host:ipc_port key [arg1..argn]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanodehost:port blockpoolId [force]]
[-setBalancerBandwidth bandwidth]
[-fetchImage local directory]
[-allowSnapshot snapshotDir]
[-disallowSnapshot snapshotDir]
[-shutdownDatanode datanode_host:ipc_port [upgrade]]
[-getDatanodeInfo datanode_host:ipc_port
[-help [cmd]
{code}
These two outputs should be the same.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6831) Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
[
https://issues.apache.org/jira/browse/HDFS-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113490#comment-14113490
]
Akira AJISAKA commented on HDFS-6831:
-
Thanks [~xiaoyuyao] for the patch!
Would you change hdfs DFSAdmin to hdfs dfsadmin?
Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
---
Key: HDFS-6831
URL: https://issues.apache.org/jira/browse/HDFS-6831
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: HDFS-6831.0.patch
There is an inconsistency between the console outputs of 'hdfs dfsadmin'
command and 'hdfs dfsadmin -help' command.
{code}
[root@trunk ~]# hdfs dfsadmin
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot snapshotDir]
[-disallowSnapshot snapshotDir]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-rollingUpgrade [query|prepare|finalize]]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota quota dirname...dirname]
[-clrQuota dirname...dirname]
[-setSpaceQuota quota dirname...dirname]
[-clrSpaceQuota dirname...dirname]
[-setBalancerBandwidth bandwidth in bytes per second]
[-fetchImage local directory]
[-shutdownDatanode datanode_host:ipc_port [upgrade]]
[-getDatanodeInfo datanode_host:ipc_port]
[-help [cmd]]
{code}
{code}
[root@trunk ~]# hdfs dfsadmin -help
hadoop dfsadmin performs DFS administrative commands.
The full syntax is:
hadoop dfsadmin
[-report [-live] [-dead] [-decommissioning]]
[-safemode enter | leave | get | wait]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-setQuota quota dirname...dirname]
[-clrQuota dirname...dirname]
[-setSpaceQuota quota dirname...dirname]
[-clrSpaceQuota dirname...dirname]
[-finalizeUpgrade]
[-rollingUpgrade [query|prepare|finalize]]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh host:ipc_port key [arg1..argn]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanodehost:port blockpoolId [force]]
[-setBalancerBandwidth bandwidth]
[-fetchImage local directory]
[-allowSnapshot snapshotDir]
[-disallowSnapshot snapshotDir]
[-shutdownDatanode datanode_host:ipc_port [upgrade]]
[-getDatanodeInfo datanode_host:ipc_port
[-help [cmd]
{code}
These two outputs should be the same.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6831) Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
[
https://issues.apache.org/jira/browse/HDFS-6831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-6831:
-
Attachment: HDFS-6831.1.patch
Update hdfs DFSAdmin to hdfs dfsadmin
Inconsistency between 'hdfs dfsadmin' and 'hdfs dfsadmin -help'
---
Key: HDFS-6831
URL: https://issues.apache.org/jira/browse/HDFS-6831
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Priority: Minor
Labels: newbie
Attachments: HDFS-6831.0.patch, HDFS-6831.1.patch
There is an inconsistency between the console outputs of 'hdfs dfsadmin'
command and 'hdfs dfsadmin -help' command.
{code}
[root@trunk ~]# hdfs dfsadmin
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot snapshotDir]
[-disallowSnapshot snapshotDir]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-rollingUpgrade [query|prepare|finalize]]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota quota dirname...dirname]
[-clrQuota dirname...dirname]
[-setSpaceQuota quota dirname...dirname]
[-clrSpaceQuota dirname...dirname]
[-setBalancerBandwidth bandwidth in bytes per second]
[-fetchImage local directory]
[-shutdownDatanode datanode_host:ipc_port [upgrade]]
[-getDatanodeInfo datanode_host:ipc_port]
[-help [cmd]]
{code}
{code}
[root@trunk ~]# hdfs dfsadmin -help
hadoop dfsadmin performs DFS administrative commands.
The full syntax is:
hadoop dfsadmin
[-report [-live] [-dead] [-decommissioning]]
[-safemode enter | leave | get | wait]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-setQuota quota dirname...dirname]
[-clrQuota dirname...dirname]
[-setSpaceQuota quota dirname...dirname]
[-clrSpaceQuota dirname...dirname]
[-finalizeUpgrade]
[-rollingUpgrade [query|prepare|finalize]]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-refreshCallQueue]
[-refresh host:ipc_port key [arg1..argn]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanodehost:port blockpoolId [force]]
[-setBalancerBandwidth bandwidth]
[-fetchImage local directory]
[-allowSnapshot snapshotDir]
[-disallowSnapshot snapshotDir]
[-shutdownDatanode datanode_host:ipc_port [upgrade]]
[-getDatanodeInfo datanode_host:ipc_port
[-help [cmd]
{code}
These two outputs should be the same.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6840) Clients are always sent to the same datanode when read is off rack
[ https://issues.apache.org/jira/browse/HDFS-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6840: -- Status: Patch Available (was: Open) Clients are always sent to the same datanode when read is off rack -- Key: HDFS-6840 URL: https://issues.apache.org/jira/browse/HDFS-6840 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Andrew Wang Priority: Critical Attachments: hdfs-6840.001.patch After HDFS-6268 the sorting order of block locations is deterministic for a given block and locality level (e.g.: local, rack. off-rack), so off-rack clients all see the same datanode for the same block. This leads to very poor behavior in distributed cache localization and other scenarios where many clients all want the same block data at approximately the same time. The one datanode is crushed by the load while the other replicas only handle local and rack-local requests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6840) Clients are always sent to the same datanode when read is off rack
[ https://issues.apache.org/jira/browse/HDFS-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6840: -- Attachment: hdfs-6840.001.patch Here's a patch that removes the random seed stuff. Fortunately HDFS-6701 hasn't been released yet, so we don't need to do any config key deprecation. branch-2 is probably going to need a new patch like the others. This might be a good time to try and figure out why we have those extra test files. Clients are always sent to the same datanode when read is off rack -- Key: HDFS-6840 URL: https://issues.apache.org/jira/browse/HDFS-6840 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Andrew Wang Priority: Critical Attachments: hdfs-6840.001.patch After HDFS-6268 the sorting order of block locations is deterministic for a given block and locality level (e.g.: local, rack. off-rack), so off-rack clients all see the same datanode for the same block. This leads to very poor behavior in distributed cache localization and other scenarios where many clients all want the same block data at approximately the same time. The one datanode is crushed by the load while the other replicas only handle local and rack-local requests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113546#comment-14113546 ] Yi Liu commented on HDFS-3529: -- Hi [~scurrilous], could I take this JIRA and continue the work? Currently HDFS cache is supported, and memory support is ongoing, so I think this will be a good improvement. Of course if use HDD, the benefit we get from saving bytes copy by using direct buffer is very small compared to the Disk IO. Use direct buffers for data in write path - Key: HDFS-3529 URL: https://issues.apache.org/jira/browse/HDFS-3529 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, performance Affects Versions: 2.0.0-alpha Reporter: Todd Lipcon Assignee: Trevor Robinson Attachments: HDFS-3529.patch, dfsio-x86-trunk-vs-3529.png The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
[
https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113551#comment-14113551
]
Hadoop QA commented on HDFS-3529:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12552318/dfsio-x86-trunk-vs-3529.png
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7808//console
This message is automatically generated.
Use direct buffers for data in write path
-
Key: HDFS-3529
URL: https://issues.apache.org/jira/browse/HDFS-3529
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, performance
Affects Versions: 2.0.0-alpha
Reporter: Todd Lipcon
Assignee: Trevor Robinson
Attachments: HDFS-3529.patch, dfsio-x86-trunk-vs-3529.png
The write path currently makes several unnecessary data copies in order to go
to and from byte arrays. We can improve performance by using direct byte
buffers to avoid the copy. This is also a prerequisite for native checksum
calculation (HDFS-3528)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6840) Clients are always sent to the same datanode when read is off rack
[
https://issues.apache.org/jira/browse/HDFS-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113557#comment-14113557
]
Hadoop QA commented on HDFS-6840:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664852/hdfs-6840.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7807//console
This message is automatically generated.
Clients are always sent to the same datanode when read is off rack
--
Key: HDFS-6840
URL: https://issues.apache.org/jira/browse/HDFS-6840
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Jason Lowe
Assignee: Andrew Wang
Priority: Critical
Attachments: hdfs-6840.001.patch
After HDFS-6268 the sorting order of block locations is deterministic for a
given block and locality level (e.g.: local, rack. off-rack), so off-rack
clients all see the same datanode for the same block. This leads to very
poor behavior in distributed cache localization and other scenarios where
many clients all want the same block data at approximately the same time.
The one datanode is crushed by the load while the other replicas only handle
local and rack-local requests.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
[
https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113583#comment-14113583
]
Hadoop QA commented on HDFS-3529:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7810//console
This message is automatically generated.
Use direct buffers for data in write path
-
Key: HDFS-3529
URL: https://issues.apache.org/jira/browse/HDFS-3529
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, performance
Affects Versions: 2.0.0-alpha
Reporter: Todd Lipcon
Assignee: Trevor Robinson
Attachments: HDFS-3529.patch, dfsio-x86-trunk-vs-3529.png
The write path currently makes several unnecessary data copies in order to go
to and from byte arrays. We can improve performance by using direct byte
buffers to avoid the copy. This is also a prerequisite for native checksum
calculation (HDFS-3528)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
[
https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113613#comment-14113613
]
Hadoop QA commented on HDFS-3529:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7813//console
This message is automatically generated.
Use direct buffers for data in write path
-
Key: HDFS-3529
URL: https://issues.apache.org/jira/browse/HDFS-3529
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, performance
Affects Versions: 2.0.0-alpha
Reporter: Todd Lipcon
Assignee: Trevor Robinson
Attachments: HDFS-3529.patch, dfsio-x86-trunk-vs-3529.png
The write path currently makes several unnecessary data copies in order to go
to and from byte arrays. We can improve performance by using direct byte
buffers to avoid the copy. This is also a prerequisite for native checksum
calculation (HDFS-3528)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
[
https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113624#comment-14113624
]
Hadoop QA commented on HDFS-3529:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7814//console
This message is automatically generated.
Use direct buffers for data in write path
-
Key: HDFS-3529
URL: https://issues.apache.org/jira/browse/HDFS-3529
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, performance
Affects Versions: 2.0.0-alpha
Reporter: Todd Lipcon
Assignee: Trevor Robinson
Attachments: HDFS-3529.patch, dfsio-x86-trunk-vs-3529.png
The write path currently makes several unnecessary data copies in order to go
to and from byte arrays. We can improve performance by using direct byte
buffers to avoid the copy. This is also a prerequisite for native checksum
calculation (HDFS-3528)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
[
https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113626#comment-14113626
]
Hadoop QA commented on HDFS-3529:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7815//console
This message is automatically generated.
Use direct buffers for data in write path
-
Key: HDFS-3529
URL: https://issues.apache.org/jira/browse/HDFS-3529
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, performance
Affects Versions: 2.0.0-alpha
Reporter: Todd Lipcon
Assignee: Trevor Robinson
Attachments: HDFS-3529.patch, dfsio-x86-trunk-vs-3529.png
The write path currently makes several unnecessary data copies in order to go
to and from byte arrays. We can improve performance by using direct byte
buffers to avoid the copy. This is also a prerequisite for native checksum
calculation (HDFS-3528)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
[
https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113627#comment-14113627
]
Hadoop QA commented on HDFS-3529:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7818//console
This message is automatically generated.
Use direct buffers for data in write path
-
Key: HDFS-3529
URL: https://issues.apache.org/jira/browse/HDFS-3529
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, performance
Affects Versions: 2.0.0-alpha
Reporter: Todd Lipcon
Assignee: Trevor Robinson
Attachments: HDFS-3529.patch, dfsio-x86-trunk-vs-3529.png
The write path currently makes several unnecessary data copies in order to go
to and from byte arrays. We can improve performance by using direct byte
buffers to avoid the copy. This is also a prerequisite for native checksum
calculation (HDFS-3528)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
[
https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113631#comment-14113631
]
Hadoop QA commented on HDFS-3529:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7819//console
This message is automatically generated.
Use direct buffers for data in write path
-
Key: HDFS-3529
URL: https://issues.apache.org/jira/browse/HDFS-3529
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, performance
Affects Versions: 2.0.0-alpha
Reporter: Todd Lipcon
Assignee: Trevor Robinson
Attachments: HDFS-3529.patch, dfsio-x86-trunk-vs-3529.png
The write path currently makes several unnecessary data copies in order to go
to and from byte arrays. We can improve performance by using direct byte
buffers to avoid the copy. This is also a prerequisite for native checksum
calculation (HDFS-3528)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6705) Create an XAttr that disallows the HDFS admin from accessing a file
[
https://issues.apache.org/jira/browse/HDFS-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113688#comment-14113688
]
Charles Lamb commented on HDFS-6705:
[~hitliuyi],
{quote}
if the super user is the owner, then it can't access the file?
{quote}
Yes, that's the intended behavior.
{quote}
It is settable by any user which has hdfs access to that file.
It can only be set and never removed.
Then any user who has hdfs access can easily prevent HDFS admin to access file
and the admin can't access that file any more. Could we find a better way?
{quote}
The xattr only prevents read access to the contents and is specifically
intended to prevent the superuser from accessing the file. I realize that the
current patch prevents read and write access to the file. Based on
[~andrew.wang]'s comments, I'm going to change it to only restrict read access,
not write access. The superuser will still have access to the meta data
(xattrs, perms, etc.) so they can chown, chmod, etc.
It's unfortunate that the superuser could lock themselves out of a file, but it
needs to be that way or else the root could just chown a users file to root and
then read it.
Create an XAttr that disallows the HDFS admin from accessing a file
---
Key: HDFS-6705
URL: https://issues.apache.org/jira/browse/HDFS-6705
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode, security
Affects Versions: 3.0.0
Reporter: Charles Lamb
Assignee: Charles Lamb
Attachments: HDFS-6705.001.patch
There needs to be an xattr that specifies that the HDFS admin can not access
a file. This is needed for m/r delegation tokens and data at rest encryption.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6705) Create an XAttr that disallows the HDFS admin from accessing a file
[ https://issues.apache.org/jira/browse/HDFS-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113690#comment-14113690 ] Charles Lamb commented on HDFS-6705: bq. I think, restricting the file not accessible by the admin will create the problem for Fsck. Ideally a Super user means, will have access to everything right? Can you give more details about this requirement.? Hi [~vinayrpet], In general transparent encryption (HDFS-6134) is intended to ensure that encrypted files are not accessible to the superuser. For MR jobs that are accessing encrypted files, the DTs need to be made unreadable to the root. The intent is that DTs will have this xattr set for this purpose. Create an XAttr that disallows the HDFS admin from accessing a file --- Key: HDFS-6705 URL: https://issues.apache.org/jira/browse/HDFS-6705 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6705.001.patch There needs to be an xattr that specifies that the HDFS admin can not access a file. This is needed for m/r delegation tokens and data at rest encryption. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6840) Clients are always sent to the same datanode when read is off rack
[
https://issues.apache.org/jira/browse/HDFS-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113720#comment-14113720
]
Yi Liu commented on HDFS-6840:
--
Thanks [~andrew.wang], the patch looks good to me. Just one thing, we need
also remove the configuration from hdfs-default.xml
{code}
property
namedfs.namenode.randomize-block-locations-per-block/name
valuefalse/value
descriptionWhen fetching replica locations of a block, the replicas
are sorted based on network distance. This configuration parameter
determines whether the replicas at the same network distance are randomly
shuffled. By default, this is false, such that repeated requests for a
block's
replicas always result in the same order. This potentially improves page
cache
behavior. However, for some network topologies, it is desirable to shuffle
this
order for better load balancing.
/description
/property
{code}
Clients are always sent to the same datanode when read is off rack
--
Key: HDFS-6840
URL: https://issues.apache.org/jira/browse/HDFS-6840
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Jason Lowe
Assignee: Andrew Wang
Priority: Critical
Attachments: hdfs-6840.001.patch
After HDFS-6268 the sorting order of block locations is deterministic for a
given block and locality level (e.g.: local, rack. off-rack), so off-rack
clients all see the same datanode for the same block. This leads to very
poor behavior in distributed cache localization and other scenarios where
many clients all want the same block data at approximately the same time.
The one datanode is crushed by the load while the other replicas only handle
local and rack-local requests.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6658) Namenode memory optimization - Block replicas list
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amir Langer updated HDFS-6658: -- Attachment: BlockListOptimizationComparison.xlsx Added comparison of memory with and without CompressedOops in both the original and modified code. (Memory in bytes collected using jmap). Difference with compressed oops is marginal but significant without it. Namenode memory optimization - Block replicas list --- Key: HDFS-6658 URL: https://issues.apache.org/jira/browse/HDFS-6658 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.4.1 Reporter: Amir Langer Assignee: Amir Langer Attachments: BlockListOptimizationComparison.xlsx, Namenode Memory Optimizations - Block replicas list.docx Part of the memory consumed by every BlockInfo object in the Namenode is a linked list of block references for every DatanodeStorageInfo (called triplets). We propose to change the way we store the list in memory. Using primitive integer indexes instead of object references will reduce the memory needed for every block replica (when compressed oops is disabled) and in our new design the list overhead will be per DatanodeStorageInfo and not per block replica. see attached design doc. for details and evaluation results. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6843) Create FSDataInputStream FSDataOutputStream isEncrypted() method
[ https://issues.apache.org/jira/browse/HDFS-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113765#comment-14113765 ] Charles Lamb commented on HDFS-6843: The test failure appears to be unrelated. I ran the test on my local machine and it passed. Create FSDataInputStream FSDataOutputStream isEncrypted() method -- Key: HDFS-6843 URL: https://issues.apache.org/jira/browse/HDFS-6843 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6843.001.patch, HDFS-6843.002.patch FileStatus should have a 'boolean isEncrypted()' method. (it was in the context of discussing with AndreW about FileStatus being a Writable). Having this method would allow MR JobSubmitter do the following: - BOOLEAN intermediateEncryption = false IF jobconf.contains(mr.intermidate.encryption) THEN intermediateEncryption = jobConf.getBoolean(mr.intermidate.encryption) ELSE IF (I/O)Format INSTANCEOF File(I/O)Format THEN intermediateEncryption = ANY File(I/O)Format HAS a Path with status isEncrypted()==TRUE FI jobConf.setBoolean(mr.intermidate.encryption, intermediateEncryption) FI -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6851) Flush EncryptionZoneWithId and add an id field to EncryptionZone
[ https://issues.apache.org/jira/browse/HDFS-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6851: --- Status: Patch Available (was: Open) Flush EncryptionZoneWithId and add an id field to EncryptionZone Key: HDFS-6851 URL: https://issues.apache.org/jira/browse/HDFS-6851 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6851.000.patch EncryptionZoneWithId can be flushed by moving the id field up to EncryptionZone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6851) Flush EncryptionZoneWithId and add an id field to EncryptionZone
[ https://issues.apache.org/jira/browse/HDFS-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6851: --- Status: Open (was: Patch Available) Jenkins not running. Try to kick it in the head again. Flush EncryptionZoneWithId and add an id field to EncryptionZone Key: HDFS-6851 URL: https://issues.apache.org/jira/browse/HDFS-6851 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6851.000.patch EncryptionZoneWithId can be flushed by moving the id field up to EncryptionZone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6658) Namenode memory optimization - Block replicas list
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113773#comment-14113773 ] Edward Bortnikov commented on HDFS-6658: Looks convincing Amir! Seems that everybody is +1 for this short-term improvement. Can we move on towards committing this code, while proceeding with the long-term discussion on HDFS-5477 and HDFS-6709? Namenode memory optimization - Block replicas list --- Key: HDFS-6658 URL: https://issues.apache.org/jira/browse/HDFS-6658 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.4.1 Reporter: Amir Langer Assignee: Amir Langer Attachments: BlockListOptimizationComparison.xlsx, Namenode Memory Optimizations - Block replicas list.docx Part of the memory consumed by every BlockInfo object in the Namenode is a linked list of block references for every DatanodeStorageInfo (called triplets). We propose to change the way we store the list in memory. Using primitive integer indexes instead of object references will reduce the memory needed for every block replica (when compressed oops is disabled) and in our new design the list overhead will be per DatanodeStorageInfo and not per block replica. see attached design doc. for details and evaluation results. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated HDFS-6888: -- Attachment: HDFS-6888-3.patch patch updated. Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113786#comment-14113786
]
Hadoop QA commented on HDFS-6888:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7820//console
This message is automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113795#comment-14113795 ] Chen He commented on HDFS-6888: --- It works on my mac. Is this because Hadoop repository move to git? Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113797#comment-14113797 ] Chen He commented on HDFS-6888: --- HEAD is now at c4c9a78 HADOOP-11001. Fix test-patch to work with the git repo. (kasha) Removing patchprocess/ Switched to branch 'trunk' Your branch is up-to-date with 'origin/trunk'. Current branch trunk is up to date. /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/jira: No such file or directory /bin/grep: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/jira: No such file or directory /bin/grep: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/jira: No such file or directory HDFS-6888 patch is being downloaded at Thu Aug 28 14:11:35 UTC 2014 from http://issues.apache.org /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory cp: cannot stat '/home/jenkins/buildSupport/lib/*': No such file or directory /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/smart-apply-patch.sh: line 44: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/smart-apply-patch.sh: line 100: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/dev-support/smart-apply-patch.sh: line 102: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/patchprocess/patch: No such file or directory The patch does not appear to apply with p0 to p2 rm: cannot remove '/tmp/tmp.paths.7370': No such file or directory PATCH APPLICATION FAILED Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HDFS-6954) With crypto, no native lib systems are too verbose
[
https://issues.apache.org/jira/browse/HDFS-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-6954 started by Charles Lamb.
With crypto, no native lib systems are too verbose
--
Key: HDFS-6954
URL: https://issues.apache.org/jira/browse/HDFS-6954
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Charles Lamb
Attachments: HDFS-6954.001.patch
Running commands on a machine without a native library results in:
{code}
$ bin/hdfs dfs -put /etc/hosts /tmp
14/08/27 07:16:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/27 07:16:11 WARN crypto.CryptoCodec: Crypto codec
org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
14/08/27 07:16:11 INFO hdfs.DFSClient: No KeyProvider found.
{code}
This is way too much.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6954) With crypto, no native lib systems are too verbose
[
https://issues.apache.org/jira/browse/HDFS-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-6954:
---
Target Version/s: 3.0.0, 2.6.0
Affects Version/s: 3.0.0
Status: Patch Available (was: In Progress)
With crypto, no native lib systems are too verbose
--
Key: HDFS-6954
URL: https://issues.apache.org/jira/browse/HDFS-6954
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Charles Lamb
Attachments: HDFS-6954.001.patch
Running commands on a machine without a native library results in:
{code}
$ bin/hdfs dfs -put /etc/hosts /tmp
14/08/27 07:16:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/27 07:16:11 WARN crypto.CryptoCodec: Crypto codec
org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
14/08/27 07:16:11 INFO hdfs.DFSClient: No KeyProvider found.
{code}
This is way too much.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6954) With crypto, no native lib systems are too verbose
[
https://issues.apache.org/jira/browse/HDFS-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-6954:
---
Attachment: HDFS-6954.001.patch
Attached is a patch which changes the CryptoCodec and DFSClient messages from
warn to debug. I considered changing the former to use the new
PerformanceAdvisory log sink, but that still wouldn't suppress the messages per
Allen's comment.
wrt distcp, I did not change the tools.DistCp or client.RMProxy messages as (I
believe) they were not changed by the fs-encryption checkin. I can easily
change those to debug also if that's desirable.
With crypto, no native lib systems are too verbose
--
Key: HDFS-6954
URL: https://issues.apache.org/jira/browse/HDFS-6954
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Charles Lamb
Attachments: HDFS-6954.001.patch
Running commands on a machine without a native library results in:
{code}
$ bin/hdfs dfs -put /etc/hosts /tmp
14/08/27 07:16:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/27 07:16:11 WARN crypto.CryptoCodec: Crypto codec
org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
14/08/27 07:16:11 INFO hdfs.DFSClient: No KeyProvider found.
{code}
This is way too much.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6954) With crypto, no native lib systems are too verbose
[
https://issues.apache.org/jira/browse/HDFS-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113857#comment-14113857
]
Hadoop QA commented on HDFS-6954:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7821//console
This message is automatically generated.
With crypto, no native lib systems are too verbose
--
Key: HDFS-6954
URL: https://issues.apache.org/jira/browse/HDFS-6954
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Charles Lamb
Attachments: HDFS-6954.001.patch
Running commands on a machine without a native library results in:
{code}
$ bin/hdfs dfs -put /etc/hosts /tmp
14/08/27 07:16:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/27 07:16:11 WARN crypto.CryptoCodec: Crypto codec
org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
14/08/27 07:16:11 INFO hdfs.DFSClient: No KeyProvider found.
{code}
This is way too much.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6694) TestPipelinesFailover.testPipelineRecoveryStress tests fail intermittently with various symptoms
[
https://issues.apache.org/jira/browse/HDFS-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113870#comment-14113870
]
Yongjun Zhang commented on HDFS-6694:
-
HI [~gkesavan] and [~ipv6guru],
Slave {{H9 (Build slave for Hadoop project builds : asf909.gq1.ygridcore.net)}}
still has the problem, it's num open files is 1024. Would you please help to
look into? Arpit suggested rebooting the host might solve the problem, which
sounds good to me.
Thanks a lot.
TestPipelinesFailover.testPipelineRecoveryStress tests fail intermittently
with various symptoms
Key: HDFS-6694
URL: https://issues.apache.org/jira/browse/HDFS-6694
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Priority: Critical
Fix For: 2.6.0
Attachments: HDFS-6694.001.dbg.patch, HDFS-6694.001.dbg.patch,
HDFS-6694.001.dbg.patch, HDFS-6694.002.dbg.patch,
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover-output.txt,
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.txt
TestPipelinesFailover.testPipelineRecoveryStress tests fail intermittently
with various symptoms. Typical failures are described in first comment.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6954) With crypto, no native lib systems are too verbose
[
https://issues.apache.org/jira/browse/HDFS-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113874#comment-14113874
]
Hadoop QA commented on HDFS-6954:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7824//console
This message is automatically generated.
With crypto, no native lib systems are too verbose
--
Key: HDFS-6954
URL: https://issues.apache.org/jira/browse/HDFS-6954
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Charles Lamb
Attachments: HDFS-6954.001.patch
Running commands on a machine without a native library results in:
{code}
$ bin/hdfs dfs -put /etc/hosts /tmp
14/08/27 07:16:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/27 07:16:11 WARN crypto.CryptoCodec: Crypto codec
org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
14/08/27 07:16:11 INFO hdfs.DFSClient: No KeyProvider found.
{code}
This is way too much.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6954) With crypto, no native lib systems are too verbose
[
https://issues.apache.org/jira/browse/HDFS-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113882#comment-14113882
]
Hadoop QA commented on HDFS-6954:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7825//console
This message is automatically generated.
With crypto, no native lib systems are too verbose
--
Key: HDFS-6954
URL: https://issues.apache.org/jira/browse/HDFS-6954
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Charles Lamb
Attachments: HDFS-6954.001.patch
Running commands on a machine without a native library results in:
{code}
$ bin/hdfs dfs -put /etc/hosts /tmp
14/08/27 07:16:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/27 07:16:11 WARN crypto.CryptoCodec: Crypto codec
org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
14/08/27 07:16:11 INFO hdfs.DFSClient: No KeyProvider found.
{code}
This is way too much.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-695) RaidNode should read in configuration from hdfs-site.xml
[
https://issues.apache.org/jira/browse/HDFS-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113887#comment-14113887
]
Hadoop QA commented on HDFS-695:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7826//console
This message is automatically generated.
RaidNode should read in configuration from hdfs-site.xml
Key: HDFS-695
URL: https://issues.apache.org/jira/browse/HDFS-695
Project: Hadoop HDFS
Issue Type: Bug
Components: contrib/raid
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Fix For: 0.21.0
Attachments: raidConfig.txt
The RaidNode currently reads in the configuration from hadoop-*.xml. It
should read in its config parameters from hdfs-site.xml as well.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6954) With crypto, no native lib systems are too verbose
[
https://issues.apache.org/jira/browse/HDFS-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113891#comment-14113891
]
Hadoop QA commented on HDFS-6954:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7827//console
This message is automatically generated.
With crypto, no native lib systems are too verbose
--
Key: HDFS-6954
URL: https://issues.apache.org/jira/browse/HDFS-6954
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Charles Lamb
Attachments: HDFS-6954.001.patch
Running commands on a machine without a native library results in:
{code}
$ bin/hdfs dfs -put /etc/hosts /tmp
14/08/27 07:16:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/27 07:16:11 WARN crypto.CryptoCodec: Crypto codec
org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
14/08/27 07:16:11 INFO hdfs.DFSClient: No KeyProvider found.
{code}
This is way too much.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen He updated HDFS-6888: -- Attachment: HDFS-6888-3.patch retrigger QA Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-3.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-3529) Use direct buffers for data in write path
[ https://issues.apache.org/jira/browse/HDFS-3529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113917#comment-14113917 ] Todd Lipcon commented on HDFS-3529: --- Hey Yi. Please note that [~james.thomas] has been doing work recently so that native CRC32 can run against byte[] arrays instead of just against byte buffers. So, the major performance improvement enabled by this JIRA probably isn't as relevant anymore. Using direct buffers might save us a memcpy, but for small cache-resident buffer sizes, my guess is that we won't see a big boost. I'd be interested to see some results, though, if you have a prototype of the patch against trunk. Use direct buffers for data in write path - Key: HDFS-3529 URL: https://issues.apache.org/jira/browse/HDFS-3529 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, performance Affects Versions: 2.0.0-alpha Reporter: Todd Lipcon Assignee: Trevor Robinson Attachments: HDFS-3529.patch, dfsio-x86-trunk-vs-3529.png The write path currently makes several unnecessary data copies in order to go to and from byte arrays. We can improve performance by using direct byte buffers to avoid the copy. This is also a prerequisite for native checksum calculation (HDFS-3528) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6962) ACLs inheritance conflict with umaskmode
LINTE created HDFS-6962: --- Summary: ACLs inheritance conflict with umaskmode Key: HDFS-6962 URL: https://issues.apache.org/jira/browse/HDFS-6962 Project: Hadoop HDFS Issue Type: Bug Components: security Affects Versions: 2.4.1 Environment: CentOS release 6.5 (Final) Reporter: LINTE In hdfs-site.xml property namedfs.umaskmode/name value027/value /property 1/ Create a directory as superuser bash# hdfs dfs -mkdir /tmp/ACLS 2/ set default ACLs on this directory rwx access for group readwrite and user toto bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS 3/ check ACLs /tmp/ACLS/ bash# hdfs dfs -getfacl /tmp/ACLS/ # file: /tmp/ACLS # owner: hdfs # group: hadoop user::rwx group::r-x other::--- default:user::rwx default:user:toto:rwx default:group::r-x default:group:readwrite:rwx default:mask::rwx default:other::--- user::rwx | group::r-x | other::--- matches with the umaskmode defined in hdfs-site.xml, everything ok ! default:group:readwrite:rwx allow readwrite group with rwx access for inhéritance. default:user:toto:rwx allow toto user with rwx access for inhéritance. default:mask::rwx inhéritance mask is rwx, so no mask 4/ Create a subdir to test inheritance of ACL bash# hdfs dfs -mkdir /tmp/ACLS/hdfs 5/ check ACLs /tmp/ACLS/hdfs bash# hdfs dfs -getfacl /tmp/ACLS/hdfs # file: /tmp/ACLS/hdfs # owner: hdfs # group: hadoop user::rwx user:toto:rwx #effective:r-x group::r-x group:readwrite:rwx #effective:r-x mask::r-x other::--- default:user::rwx default:user:toto:rwx default:group::r-x default:group:readwrite:rwx default:mask::rwx default:other::--- Here we can see that the readwrite group has rwx ACL bu only r-x is effective because the mask is r-x (mask::r-x) in spite of default mask for inheritance is set to default:mask::rwx on /tmp/ACLS/ 6/ Modifiy hdfs-site.xml et restart namenode property namedfs.umaskmode/name value010/value /property 7/ Create a subdir to test inheritance of ACL with new parameter umaskmode bash# hdfs dfs -mkdir /tmp/ACLS/hdfs2 8/ Check ACL on /tmp/ACLS/hdfs2 bash# hdfs dfs -getfacl /tmp/ACLS/hdfs2 # file: /tmp/ACLS/hdfs2 # owner: hdfs # group: hadoop user::rwx user:toto:rwx #effective:rw- group::r-x #effective:r-- group:readwrite:rwx #effective:rw- mask::rw- other::--- default:user::rwx default:user:toto:rwx default:group::r-x default:group:readwrite:rwx default:mask::rwx default:other::--- So HDFS masks the ACL value (user, group and other -- exepted the POSIX owner -- ) with the group mask of dfs.umaskmode properties when creating directory with inherited ACL. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6867) For DFSOutputStream, do pipeline recovery for a single block in the background
[ https://issues.apache.org/jira/browse/HDFS-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-6867: Attachment: HDFS-6867-20140828-1.patch Some test classes have RepliceNodeOnFailure disabled. This new patches skips recoveryWorker invocation if that's the case. For DFSOutputStream, do pipeline recovery for a single block in the background -- Key: HDFS-6867 URL: https://issues.apache.org/jira/browse/HDFS-6867 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Colin Patrick McCabe Assignee: Zhe Zhang Attachments: HDFS-6867-20140827-2.patch, HDFS-6867-20140827-3.patch, HDFS-6867-20140827.patch, HDFS-6867-20140828-1.patch, HDFS-6867-design-20140820.pdf, HDFS-6867-design-20140821.pdf, HDFS-6867-design-20140822.pdf, HDFS-6867-design-20140827.pdf For DFSOutputStream, we should be able to do pipeline recovery in the background, while the user is continuing to write to the file. This is especially useful for long-lived clients that write to an HDFS file slowly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6958) Fix the condition of reporting failures in BlockPoolSlice
[
https://issues.apache.org/jira/browse/HDFS-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113961#comment-14113961
]
Lei (Eddy) Xu commented on HDFS-6958:
-
[~atm] These two failures are all related to balancer, which is not related to
this patch.
Fix the condition of reporting failures in BlockPoolSlice
-
Key: HDFS-6958
URL: https://issues.apache.org/jira/browse/HDFS-6958
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Trivial
Attachments: HDFS-6958.000.patch
{{BlockPoolSlice#addToReplicsMap()}} should log the failure for the operation
that deletes restart metadata file.
The current code did the opposite, log the failures on succeed deletion:
{code:java}
if (restartMeta.delete()) {
FsDatasetImpl.LOG.warn(Failed to delete restart meta file: +
restartMeta.getPath());
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6963) Test partial failures when loading / removing volumes.
Lei (Eddy) Xu created HDFS-6963: --- Summary: Test partial failures when loading / removing volumes. Key: HDFS-6963 URL: https://issues.apache.org/jira/browse/HDFS-6963 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: 2.5.0 Reporter: Lei (Eddy) Xu Assignee: Lei (Eddy) Xu Test the cases that there are partial failures during loading new volumes. The expected behavior should be trying the best efforts to successfully load all good volumes, and then report the failures. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6964) NN fails to fix under replication leading to data loss
Daryn Sharp created HDFS-6964: - Summary: NN fails to fix under replication leading to data loss Key: HDFS-6964 URL: https://issues.apache.org/jira/browse/HDFS-6964 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Priority: Blocker We've encountered lost blocks due to node failure even when there is ample time to fix the under-replication. 2 nodes were lost. The 3rd node with the last remaining replicas averaged 1 copy block per heartbeat (3s) until ~7h later when that node was lost resulting in over 50 lost blocks. When the node was restarted and sent its BR the NN immediately began fixing the replication. In another data loss event, over 150 blocks were lost due to node failure but the timing of the node loss is not known so there may have been inadequate time to fix the under-replication unlike the first case. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6965) NN continues to issue block locations for DNs with full disks
Daryn Sharp created HDFS-6965: - Summary: NN continues to issue block locations for DNs with full disks Key: HDFS-6965 URL: https://issues.apache.org/jira/browse/HDFS-6965 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Encountered issues where DNs have less space than a full block and reject incoming transfers. The NN continues giving out locations for these nodes for some period of time. It does not appear to be related to the DN's cached disk usage. One impact is required replications are delayed when a full DN is chosen for the pipeline. A DN cannot report a broken pipeline so the replication must timeout (5m) before new targets are chosen. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6965) NN continues to issue block locations for DNs with full disks
[
https://issues.apache.org/jira/browse/HDFS-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114065#comment-14114065
]
Kihwal Lee commented on HDFS-6965:
--
I think there is a flaw in isGoodTarget(). When checking the available space,
it checks against {{node.getRemaining()}}. It has too check at individual
storage level.
NN continues to issue block locations for DNs with full disks
-
Key: HDFS-6965
URL: https://issues.apache.org/jira/browse/HDFS-6965
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Encountered issues where DNs have less space than a full block and reject
incoming transfers. The NN continues giving out locations for these nodes
for some period of time. It does not appear to be related to the DN's cached
disk usage.
One impact is required replications are delayed when a full DN is chosen for
the pipeline. A DN cannot report a broken pipeline so the replication must
timeout (5m) before new targets are chosen.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6843) Create FSDataInputStream FSDataOutputStream isEncrypted() method
[ https://issues.apache.org/jira/browse/HDFS-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114075#comment-14114075 ] Steve Loughran commented on HDFS-6843: -- As this is extending the core FS APIs # Can you update the specification of those FS APIs to add a new attribute to a file status # the contract tests need to be extended too, we just need to verify that isEncrypted returns something ... that could be integrated with one of the exiting tests This may need to be mentioned in a HADOOP- JIRA even if the patch is in here ... this is broader than just HDFS Create FSDataInputStream FSDataOutputStream isEncrypted() method -- Key: HDFS-6843 URL: https://issues.apache.org/jira/browse/HDFS-6843 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6843.001.patch, HDFS-6843.002.patch FileStatus should have a 'boolean isEncrypted()' method. (it was in the context of discussing with AndreW about FileStatus being a Writable). Having this method would allow MR JobSubmitter do the following: - BOOLEAN intermediateEncryption = false IF jobconf.contains(mr.intermidate.encryption) THEN intermediateEncryption = jobConf.getBoolean(mr.intermidate.encryption) ELSE IF (I/O)Format INSTANCEOF File(I/O)Format THEN intermediateEncryption = ANY File(I/O)Format HAS a Path with status isEncrypted()==TRUE FI jobConf.setBoolean(mr.intermidate.encryption, intermediateEncryption) FI -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6966) Add additional unit tests for encryption zones
Stephen Chu created HDFS-6966: - Summary: Add additional unit tests for encryption zones Key: HDFS-6966 URL: https://issues.apache.org/jira/browse/HDFS-6966 Project: Hadoop HDFS Issue Type: Sub-task Components: encryption Affects Versions: 3.0.0 Reporter: Stephen Chu Assignee: Stephen Chu There are some more unit tests that can be added for test encryption zones. For example, more encryption zone + snapshot tests, running fsck on encryption zones, and more. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6946) TestBalancerWithSaslDataTransfer fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114095#comment-14114095
]
Stephen Chu commented on HDFS-6946:
---
[~airbots], [~kihwal], any thoughts on this one? Checking because you worked on
HDFS-5803.
TestBalancerWithSaslDataTransfer fails in trunk
---
Key: HDFS-6946
URL: https://issues.apache.org/jira/browse/HDFS-6946
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Stephen Chu
Priority: Minor
Attachments: HDFS-6946.1.patch
From build #1849 :
{code}
REGRESSION:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer.testBalancer0Integrity
Error Message:
Cluster failed to reached expected values of totalSpace (current: 750,
expected: 750), or usedSpace (current: 140, expected: 150), in more than
4 msec.
Stack Trace:
java.util.concurrent.TimeoutException: Cluster failed to reached expected
values of totalSpace (current: 750, expected: 750), or usedSpace (current:
140, expected: 150), in more than 4 msec.
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForHeartBeat(TestBalancer.java:253)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:578)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:551)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:437)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:645)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0Internal(TestBalancer.java:759)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer.testBalancer0Integrity(TestBalancerWithSaslDataTransfer.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6967) DNs may OOM under high webhdfs load
Daryn Sharp created HDFS-6967: - Summary: DNs may OOM under high webhdfs load Key: HDFS-6967 URL: https://issues.apache.org/jira/browse/HDFS-6967 Project: Hadoop HDFS Issue Type: Bug Components: datanode, webhdfs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Webhdfs uses jetty. The size of the request thread pool is limited, but jetty will accept and queue infinite connections. Every queued connection is heavy with buffers, etc. Unlike data streamer connections, thousands of webhdfs connections will quickly OOM a DN. The accepted requests must be bounded and excess clients rejected so they retry on a new DN. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[
https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114141#comment-14114141
]
Hadoop QA commented on HDFS-6888:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664998/HDFS-6888-3.patch
against trunk revision c4c9a78.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.security.TestRefreshUserMappings
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDecommission
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7829//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7829//console
This message is automatically generated.
Remove audit logging of getFIleInfo()
-
Key: HDFS-6888
URL: https://issues.apache.org/jira/browse/HDFS-6888
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Kihwal Lee
Assignee: Chen He
Labels: log
Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-3.patch,
HDFS-6888.patch
The audit logging of getFileInfo() was added in HDFS-3733. Since this is a
one of the most called method, users have noticed that audit log is now
filled with this. Since we now have HTTP request logging, this seems
unnecessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[ https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114137#comment-14114137 ] Jitendra Nath Pandey commented on HDFS-6826: I do prefer v7.4 approach. I believe plugins should have the flexibility to exclusively control the permission check. Plugin interface to enable delegation of HDFS authorization assertions -- Key: HDFS-6826 URL: https://issues.apache.org/jira/browse/HDFS-6826 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.4.1 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch, HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch, HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.patch, HDFS-6826v8.patch, HDFSPluggableAuthorizationProposal-v2.pdf, HDFSPluggableAuthorizationProposal.pdf When Hbase data, HiveMetaStore data or Search data is accessed via services (Hbase region servers, HiveServer2, Impala, Solr) the services can enforce permissions on corresponding entities (databases, tables, views, columns, search collections, documents). It is desirable, when the data is accessed directly by users accessing the underlying data files (i.e. from a MapReduce job), that the permission of the data files map to the permissions of the corresponding data entity (i.e. table, column family or search collection). To enable this we need to have the necessary hooks in place in the NameNode to delegate authorization to an external system that can map HDFS files/directories to data entities and resolve their permissions based on the data entities permissions. I’ll be posting a design proposal in the next few days. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6954) With crypto, no native lib systems are too verbose
[
https://issues.apache.org/jira/browse/HDFS-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114140#comment-14114140
]
Hadoop QA commented on HDFS-6954:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12664990/HDFS-6954.001.patch
against trunk revision c4c9a78.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The applied patch generated 1263 javac
compiler warnings (more than the trunk's current 802 warnings).
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 9
warning messages.
See
https://builds.apache.org/job/PreCommit-HDFS-Build/7828//artifact/trunk/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDecommission
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7828//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7828//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7828//console
This message is automatically generated.
With crypto, no native lib systems are too verbose
--
Key: HDFS-6954
URL: https://issues.apache.org/jira/browse/HDFS-6954
Project: Hadoop HDFS
Issue Type: Bug
Components: encryption
Affects Versions: 3.0.0
Reporter: Allen Wittenauer
Assignee: Charles Lamb
Attachments: HDFS-6954.001.patch
Running commands on a machine without a native library results in:
{code}
$ bin/hdfs dfs -put /etc/hosts /tmp
14/08/27 07:16:10 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/27 07:16:11 WARN crypto.CryptoCodec: Crypto codec
org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec is not available.
14/08/27 07:16:11 INFO hdfs.DFSClient: No KeyProvider found.
{code}
This is way too much.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-4257) The ReplaceDatanodeOnFailure policies could have a forgiving option
[
https://issues.apache.org/jira/browse/HDFS-4257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114150#comment-14114150
]
Yongjun Zhang commented on HDFS-4257:
-
HI Nicholas, I wonder if we could log a message to see what policy is used at
initialization stage and whenever the policy is changed, so when we check the
log, we can see what policy is being used? thanks.
The ReplaceDatanodeOnFailure policies could have a forgiving option
---
Key: HDFS-4257
URL: https://issues.apache.org/jira/browse/HDFS-4257
Project: Hadoop HDFS
Issue Type: New Feature
Components: hdfs-client
Affects Versions: 2.0.2-alpha
Reporter: Harsh J
Assignee: Tsz Wo Nicholas Sze
Priority: Minor
Attachments: h4257_20140325.patch, h4257_20140325b.patch,
h4257_20140326.patch, h4257_20140819.patch
Similar question has previously come over HDFS-3091 and friends, but the
essential problem is: Why can't I write to my cluster of 3 nodes, when I
just have 1 node available at a point in time..
The policies cover the 4 options, with {{Default}} being default:
{{Disable}} - Disables the whole replacement concept by throwing out an
error (at the server) or acts as {{Never}} at the client.
{{Never}} - Never replaces a DN upon pipeline failures (not too desirable in
many cases).
{{Default}} - Replace based on a few conditions, but whose minimum never
touches 1. We always fail if only one DN remains and none others can be added.
{{Always}} - Replace no matter what. Fail if can't replace.
Would it not make sense to have an option similar to Always/Default, where
despite _trying_, if it isn't possible to have 1 DN in the pipeline, do not
fail. I think that is what the former write behavior was, and what fit with
the minimum replication factor allowed value.
Why is it grossly wrong to pass a write from a client for a block with just 1
remaining replica in the pipeline (the minimum of 1 grows with the
replication factor demanded from the write), when replication is taken care
of immediately afterwards? How often have we seen missing blocks arise out of
allowing this + facing a big rack(s) failure or so?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6928) 'hdfs put' command should accept lazyPersist flag for testing
[ https://issues.apache.org/jira/browse/HDFS-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114176#comment-14114176 ] Xiaoyu Yao commented on HDFS-6928: -- Looks good. Can you update the comment of with lazyPersist flag on the stat Format sequences in Stat.java? 'hdfs put' command should accept lazyPersist flag for testing - Key: HDFS-6928 URL: https://issues.apache.org/jira/browse/HDFS-6928 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: HDFS-6581 Reporter: Tassapol Athiapinya Assignee: Arpit Agarwal Attachments: HDFS-6928.01.patch, HDFS-6928.02.patch Add a '-l' flag to 'hdfs put' which creates the file with the LAZY_PERSIST option. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6843) Create FSDataInputStream FSDataOutputStream isEncrypted() method
[ https://issues.apache.org/jira/browse/HDFS-6843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114192#comment-14114192 ] jay vyas commented on HDFS-6843: as per [~ste...@apache.org]'s comment, definetly important to put this specification in HADOOP-... so that the other HCFS implementations can move it into the contract, so we can have unit tests for it, etc, using . Create FSDataInputStream FSDataOutputStream isEncrypted() method -- Key: HDFS-6843 URL: https://issues.apache.org/jira/browse/HDFS-6843 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Affects Versions: 3.0.0 Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6843.001.patch, HDFS-6843.002.patch FileStatus should have a 'boolean isEncrypted()' method. (it was in the context of discussing with AndreW about FileStatus being a Writable). Having this method would allow MR JobSubmitter do the following: - BOOLEAN intermediateEncryption = false IF jobconf.contains(mr.intermidate.encryption) THEN intermediateEncryption = jobConf.getBoolean(mr.intermidate.encryption) ELSE IF (I/O)Format INSTANCEOF File(I/O)Format THEN intermediateEncryption = ANY File(I/O)Format HAS a Path with status isEncrypted()==TRUE FI jobConf.setBoolean(mr.intermidate.encryption, intermediateEncryption) FI -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6946) TestBalancerWithSaslDataTransfer fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114194#comment-14114194
]
Chen He commented on HDFS-6946:
---
[~schu] can you provide the full log if this error? Balancer test fails because
of many reasons and some errors happen intermittently.
TestBalancerWithSaslDataTransfer fails in trunk
---
Key: HDFS-6946
URL: https://issues.apache.org/jira/browse/HDFS-6946
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Stephen Chu
Priority: Minor
Attachments: HDFS-6946.1.patch
From build #1849 :
{code}
REGRESSION:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer.testBalancer0Integrity
Error Message:
Cluster failed to reached expected values of totalSpace (current: 750,
expected: 750), or usedSpace (current: 140, expected: 150), in more than
4 msec.
Stack Trace:
java.util.concurrent.TimeoutException: Cluster failed to reached expected
values of totalSpace (current: 750, expected: 750), or usedSpace (current:
140, expected: 150), in more than 4 msec.
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForHeartBeat(TestBalancer.java:253)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:578)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:551)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:437)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:645)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0Internal(TestBalancer.java:759)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer.testBalancer0Integrity(TestBalancerWithSaslDataTransfer.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6967) DNs may OOM under high webhdfs load
[ https://issues.apache.org/jira/browse/HDFS-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-6967: -- Assignee: Eric Payne DNs may OOM under high webhdfs load --- Key: HDFS-6967 URL: https://issues.apache.org/jira/browse/HDFS-6967 Project: Hadoop HDFS Issue Type: Bug Components: datanode, webhdfs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Eric Payne Webhdfs uses jetty. The size of the request thread pool is limited, but jetty will accept and queue infinite connections. Every queued connection is heavy with buffers, etc. Unlike data streamer connections, thousands of webhdfs connections will quickly OOM a DN. The accepted requests must be bounded and excess clients rejected so they retry on a new DN. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114200#comment-14114200 ] Chen He commented on HDFS-6888: --- Test error is because of HADOOP-11020 Remove audit logging of getFIleInfo() - Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Kihwal Lee Assignee: Chen He Labels: log Attachments: HDFS-6888-2.patch, HDFS-6888-3.patch, HDFS-6888-3.patch, HDFS-6888.patch The audit logging of getFileInfo() was added in HDFS-3733. Since this is a one of the most called method, users have noticed that audit log is now filled with this. Since we now have HTTP request logging, this seems unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6946) TestBalancerWithSaslDataTransfer fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Chu updated HDFS-6946:
--
Attachment: testBalancer0Integrity-failure.log
Attaching testBalancer0Integrity failure log from jenkins report. Will go
through them again too.
TestBalancerWithSaslDataTransfer fails in trunk
---
Key: HDFS-6946
URL: https://issues.apache.org/jira/browse/HDFS-6946
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Stephen Chu
Priority: Minor
Attachments: HDFS-6946.1.patch, testBalancer0Integrity-failure.log
From build #1849 :
{code}
REGRESSION:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer.testBalancer0Integrity
Error Message:
Cluster failed to reached expected values of totalSpace (current: 750,
expected: 750), or usedSpace (current: 140, expected: 150), in more than
4 msec.
Stack Trace:
java.util.concurrent.TimeoutException: Cluster failed to reached expected
values of totalSpace (current: 750, expected: 750), or usedSpace (current:
140, expected: 150), in more than 4 msec.
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForHeartBeat(TestBalancer.java:253)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:578)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:551)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:437)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:645)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0Internal(TestBalancer.java:759)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer.testBalancer0Integrity(TestBalancerWithSaslDataTransfer.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6946) TestBalancerWithSaslDataTransfer fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114205#comment-14114205
]
Hadoop QA commented on HDFS-6946:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665044/testBalancer0Integrity-failure.log
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7831//console
This message is automatically generated.
TestBalancerWithSaslDataTransfer fails in trunk
---
Key: HDFS-6946
URL: https://issues.apache.org/jira/browse/HDFS-6946
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Stephen Chu
Priority: Minor
Attachments: HDFS-6946.1.patch, testBalancer0Integrity-failure.log
From build #1849 :
{code}
REGRESSION:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer.testBalancer0Integrity
Error Message:
Cluster failed to reached expected values of totalSpace (current: 750,
expected: 750), or usedSpace (current: 140, expected: 150), in more than
4 msec.
Stack Trace:
java.util.concurrent.TimeoutException: Cluster failed to reached expected
values of totalSpace (current: 750, expected: 750), or usedSpace (current:
140, expected: 150), in more than 4 msec.
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForHeartBeat(TestBalancer.java:253)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:578)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:551)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:437)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:645)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0Internal(TestBalancer.java:759)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer.testBalancer0Integrity(TestBalancerWithSaslDataTransfer.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6946) TestBalancerWithSaslDataTransfer fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-6946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114210#comment-14114210
]
Chen He commented on HDFS-6946:
---
TestRefreshUserMappings errors because of HADOOP-11020
TestBalancerWithSaslDataTransfer fails in trunk
---
Key: HDFS-6946
URL: https://issues.apache.org/jira/browse/HDFS-6946
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Assignee: Stephen Chu
Priority: Minor
Attachments: HDFS-6946.1.patch, testBalancer0Integrity-failure.log
From build #1849 :
{code}
REGRESSION:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer.testBalancer0Integrity
Error Message:
Cluster failed to reached expected values of totalSpace (current: 750,
expected: 750), or usedSpace (current: 140, expected: 150), in more than
4 msec.
Stack Trace:
java.util.concurrent.TimeoutException: Cluster failed to reached expected
values of totalSpace (current: 750, expected: 750), or usedSpace (current:
140, expected: 150), in more than 4 msec.
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.waitForHeartBeat(TestBalancer.java:253)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.runBalancer(TestBalancer.java:578)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:551)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.doTest(TestBalancer.java:437)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.oneNodeTest(TestBalancer.java:645)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancer0Internal(TestBalancer.java:759)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer.testBalancer0Integrity(TestBalancerWithSaslDataTransfer.java:34)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6867) For DFSOutputStream, do pipeline recovery for a single block in the background
[
https://issues.apache.org/jira/browse/HDFS-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114207#comment-14114207
]
Hadoop QA commented on HDFS-6867:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665006/HDFS-6867-20140828-1.patch
against trunk revision c4c9a78.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.TestSafeMode
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7830//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7830//console
This message is automatically generated.
For DFSOutputStream, do pipeline recovery for a single block in the background
--
Key: HDFS-6867
URL: https://issues.apache.org/jira/browse/HDFS-6867
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Reporter: Colin Patrick McCabe
Assignee: Zhe Zhang
Attachments: HDFS-6867-20140827-2.patch, HDFS-6867-20140827-3.patch,
HDFS-6867-20140827.patch, HDFS-6867-20140828-1.patch,
HDFS-6867-design-20140820.pdf, HDFS-6867-design-20140821.pdf,
HDFS-6867-design-20140822.pdf, HDFS-6867-design-20140827.pdf
For DFSOutputStream, we should be able to do pipeline recovery in the
background, while the user is continuing to write to the file. This is
especially useful for long-lived clients that write to an HDFS file slowly.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6940) Initial refactoring to allow ConsensusNode implementation
[ https://issues.apache.org/jira/browse/HDFS-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114213#comment-14114213 ] Konstantin Boudnik commented on HDFS-6940: -- bq. I think it'd be much better if you could somehow abstract out the behavior of the NN that the ConsensusNode needs to change into some sort of plugin interface, with a default implementation just being what the NN currently does, and then you could provide an alternate implementation that does what the ConsensusNode needs to do. If I am reading this right, you'd be ok with a potentially huge refactoring of NN followed by one the two: # significant duplication of the NN code in the CNode # unnecessarily exposing the implementation of many intimate parts of NN Or would it acceptable to add a dynamic dependency injection mechanism, perhaps? Initial refactoring to allow ConsensusNode implementation - Key: HDFS-6940 URL: https://issues.apache.org/jira/browse/HDFS-6940 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.0.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko Attachments: HDFS-6940.patch Minor refactoring of FSNamesystem to open private methods that are needed for CNode implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)
[
https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114221#comment-14114221
]
stack commented on HDFS-6865:
-
bq, Stack, by any chance did you run any HBase correctness or system tests with
the patch enabled?
Ran a short test.IntegrationTestBigLinkedList on a small cluster.
{code}
2014-08-28 12:18:30,126 INFO [main] test.IntegrationTestBigLinkedList$Loop:
Verify finished with succees. Total nodes=200
{code}
I'd imagine it would fail if we were misreading.
Byte array native checksumming on client side (HDFS changes)
Key: HDFS-6865
URL: https://issues.apache.org/jira/browse/HDFS-6865
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client, performance
Reporter: James Thomas
Assignee: James Thomas
Attachments: HDFS-6865.2.patch, HDFS-6865.3.patch, HDFS-6865.4.patch,
HDFS-6865.5.patch, HDFS-6865.6.patch, HDFS-6865.7.patch, HDFS-6865.8.patch,
HDFS-6865.patch
Refactor FSOutputSummer to buffer data and use the native checksum
calculation functionality introduced in HADOOP-10975.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6951) Saving namespace and restarting NameNode will remove existing encryption zones
[
https://issues.apache.org/jira/browse/HDFS-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-6951:
---
Attachment: HDFS-6951.002.patch
[~andrew.wang], [~hitliuyi],
Thanks for your reviews and comments. I've made the changes that you
suggested. I've also run TestOfflineEditsViewer and it passes:
---
T E S T S
---
Running org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.466 sec - in
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
Results :
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
Saving namespace and restarting NameNode will remove existing encryption zones
--
Key: HDFS-6951
URL: https://issues.apache.org/jira/browse/HDFS-6951
Project: Hadoop HDFS
Issue Type: Sub-task
Components: encryption
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Charles Lamb
Attachments: HDFS-6951-prelim.002.patch, HDFS-6951-testrepo.patch,
HDFS-6951.001.patch, HDFS-6951.002.patch, editsStored
Currently, when users save namespace and restart the NameNode, pre-existing
encryption zones will be wiped out.
I could reproduce this on a pseudo-distributed cluster:
* Create an encryption zone
* List encryption zones and verify the newly created zone is present
* Save the namespace
* Kill and restart the NameNode
* List the encryption zones and you'll find the encryption zone is missing
I've attached a test case for {{TestEncryptionZones}} that reproduces this as
well. Removing the saveNamespace call will get the test to pass.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HDFS-6965) NN continues to issue block locations for DNs with full disks
[ https://issues.apache.org/jira/browse/HDFS-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li reassigned HDFS-6965: -- Assignee: chang li NN continues to issue block locations for DNs with full disks - Key: HDFS-6965 URL: https://issues.apache.org/jira/browse/HDFS-6965 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: chang li Encountered issues where DNs have less space than a full block and reject incoming transfers. The NN continues giving out locations for these nodes for some period of time. It does not appear to be related to the DN's cached disk usage. One impact is required replications are delayed when a full DN is chosen for the pipeline. A DN cannot report a broken pipeline so the replication must timeout (5m) before new targets are chosen. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6951) Saving namespace and restarting NameNode will remove existing encryption zones
[
https://issues.apache.org/jira/browse/HDFS-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-6951:
---
Attachment: editsStored
The binary editsStored file is attached.
Saving namespace and restarting NameNode will remove existing encryption zones
--
Key: HDFS-6951
URL: https://issues.apache.org/jira/browse/HDFS-6951
Project: Hadoop HDFS
Issue Type: Sub-task
Components: encryption
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Charles Lamb
Attachments: HDFS-6951-prelim.002.patch, HDFS-6951-testrepo.patch,
HDFS-6951.001.patch, HDFS-6951.002.patch, editsStored
Currently, when users save namespace and restart the NameNode, pre-existing
encryption zones will be wiped out.
I could reproduce this on a pseudo-distributed cluster:
* Create an encryption zone
* List encryption zones and verify the newly created zone is present
* Save the namespace
* Kill and restart the NameNode
* List the encryption zones and you'll find the encryption zone is missing
I've attached a test case for {{TestEncryptionZones}} that reproduces this as
well. Removing the saveNamespace call will get the test to pass.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6951) Saving namespace and restarting NameNode will remove existing encryption zones
[
https://issues.apache.org/jira/browse/HDFS-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114243#comment-14114243
]
Hadoop QA commented on HDFS-6951:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665047/editsStored
against trunk revision c4c9a78.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7832//console
This message is automatically generated.
Saving namespace and restarting NameNode will remove existing encryption zones
--
Key: HDFS-6951
URL: https://issues.apache.org/jira/browse/HDFS-6951
Project: Hadoop HDFS
Issue Type: Sub-task
Components: encryption
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Charles Lamb
Attachments: HDFS-6951-prelim.002.patch, HDFS-6951-testrepo.patch,
HDFS-6951.001.patch, HDFS-6951.002.patch, editsStored
Currently, when users save namespace and restart the NameNode, pre-existing
encryption zones will be wiped out.
I could reproduce this on a pseudo-distributed cluster:
* Create an encryption zone
* List encryption zones and verify the newly created zone is present
* Save the namespace
* Kill and restart the NameNode
* List the encryption zones and you'll find the encryption zone is missing
I've attached a test case for {{TestEncryptionZones}} that reproduces this as
well. Removing the saveNamespace call will get the test to pass.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6860) BlockStateChange logs are too noisy
[ https://issues.apache.org/jira/browse/HDFS-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chang li updated HDFS-6860: --- Attachment: HDFS6860.patch have changed most of the blockstatechange log from info level to debug level. BlockStateChange logs are too noisy --- Key: HDFS-6860 URL: https://issues.apache.org/jira/browse/HDFS-6860 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.5.0 Reporter: Arpit Agarwal Assignee: chang li Labels: newbie Attachments: HDFS6860.patch Block State Change logs are too noisy at the default INFO level and affect NN performance on busy clusters. Most of these state changes can be logged at debug level instead. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Thomas updated HDFS-6800:
---
Attachment: HDFS-6800.7.patch
Uploaded patch that addresses case 2) above. [~arpitagarwal], thoughts?
Determine how Datanode layout changes should interact with rolling upgrade
--
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6860) BlockStateChange logs are too noisy
[
https://issues.apache.org/jira/browse/HDFS-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114291#comment-14114291
]
Arpit Agarwal commented on HDFS-6860:
-
Thanks for fixing this [~lichangleo]. Can you also wrap each call in a
{{isDebugEnabled()}} call. That way we don't incur the overhead of generating
the log strings unless debug logging is turned on. e.g.
{code}
if (NameNode.blockStateChangeLog.isDebugEnabled()) {
NameNode.blockStateChangeLog.debug(BLOCK* Removing stale replica
+ from location: + r.getExpectedStorageLocation());
}
{code}
BlockStateChange logs are too noisy
---
Key: HDFS-6860
URL: https://issues.apache.org/jira/browse/HDFS-6860
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.5.0
Reporter: Arpit Agarwal
Assignee: chang li
Labels: newbie
Attachments: HDFS6860.patch
Block State Change logs are too noisy at the default INFO level and affect NN
performance on busy clusters.
Most of these state changes can be logged at debug level instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6940) Initial refactoring to allow ConsensusNode implementation
[
https://issues.apache.org/jira/browse/HDFS-6940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114294#comment-14114294
]
Aaron T. Myers commented on HDFS-6940:
--
{quote}
If I am reading this right, you'd be ok with a potentially huge refactoring of
NN followed by one the two:
1. significant duplication of the NN code in the CNode
2. unnecessarily exposing the implementation of many intimate parts of NN
{quote}
I was not necessarily suggesting any of those things. It's not at all clear to
me that said refactor would necessarily be huge, or that it would result in
any duplication of NN code in the ConsensusNode. If it's designed well I don't
see why either of those things would necessarily be the case. That very much
remains to be seen, and just asserting that it would be so is not very
constructive.
Initial refactoring to allow ConsensusNode implementation
-
Key: HDFS-6940
URL: https://issues.apache.org/jira/browse/HDFS-6940
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 3.0.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Attachments: HDFS-6940.patch
Minor refactoring of FSNamesystem to open private methods that are needed for
CNode implementation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6867) For DFSOutputStream, do pipeline recovery for a single block in the background
[ https://issues.apache.org/jira/browse/HDFS-6867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-6867: Attachment: HDFS-6867-20140828-2.patch The new patch adds a type of policy (BACKGROUND) where background recovery worker will be invoked. For DFSOutputStream, do pipeline recovery for a single block in the background -- Key: HDFS-6867 URL: https://issues.apache.org/jira/browse/HDFS-6867 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Colin Patrick McCabe Assignee: Zhe Zhang Attachments: HDFS-6867-20140827-2.patch, HDFS-6867-20140827-3.patch, HDFS-6867-20140827.patch, HDFS-6867-20140828-1.patch, HDFS-6867-20140828-2.patch, HDFS-6867-design-20140820.pdf, HDFS-6867-design-20140821.pdf, HDFS-6867-design-20140822.pdf, HDFS-6867-design-20140827.pdf For DFSOutputStream, we should be able to do pipeline recovery in the background, while the user is continuing to write to the file. This is especially useful for long-lived clients that write to an HDFS file slowly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114306#comment-14114306
]
Arpit Agarwal commented on HDFS-6800:
-
James, would it be simpler to just fix {{NameNodeRpcServer.blockReport}} to
FinalizeCommand only if {{namesystem.isRollingUpgrade}} returns false or is
there some other implication of leaving {{isUpgradeFinalized}} to false that I
am not thinking of?
Determine how Datanode layout changes should interact with rolling upgrade
--
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6860) BlockStateChange logs are too noisy
[
https://issues.apache.org/jira/browse/HDFS-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114304#comment-14114304
]
Andrew Wang commented on HDFS-6860:
---
Is this a good time to also switch BlockStateChange over to use slf4j and to
use {} templates? Then we could avoid the if-wrappers.
BlockStateChange logs are too noisy
---
Key: HDFS-6860
URL: https://issues.apache.org/jira/browse/HDFS-6860
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.5.0
Reporter: Arpit Agarwal
Assignee: chang li
Labels: newbie
Attachments: HDFS6860.patch
Block State Change logs are too noisy at the default INFO level and affect NN
performance on busy clusters.
Most of these state changes can be logged at debug level instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114306#comment-14114306
]
Arpit Agarwal edited comment on HDFS-6800 at 8/28/14 8:38 PM:
--
James, would it be simpler to just fix {{NameNodeRpcServer.blockReport}} to
return FinalizeCommand only if {{namesystem.isRollingUpgrade}} returns false or
is there some other implication of leaving {{isUpgradeFinalized}} to false that
I am not thinking of?
was (Author: arpitagarwal):
James, would it be simpler to just fix {{NameNodeRpcServer.blockReport}} to
FinalizeCommand only if {{namesystem.isRollingUpgrade}} returns false or is
there some other implication of leaving {{isUpgradeFinalized}} to false that I
am not thinking of?
Determine how Datanode layout changes should interact with rolling upgrade
--
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6968) Configurable replication degree in the hadoop archive command
Zhe Zhang created HDFS-6968:
---
Summary: Configurable replication degree in the hadoop archive
command
Key: HDFS-6968
URL: https://issues.apache.org/jira/browse/HDFS-6968
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Due to be below hard-coded replication degree in {{HadoopArchives}}, the
{{archive}} command will fail if HDFS maximum replication has already been
configured to a number lower than 10.
{code:java}
//increase the replication of src files
jobfs.setReplication(srcFiles, (short) 10);
{code}
This Jira will make the {{archive}} command configurable with desired
replication degree.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6860) BlockStateChange logs are too noisy
[
https://issues.apache.org/jira/browse/HDFS-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114327#comment-14114327
]
Arpit Agarwal commented on HDFS-6860:
-
[~lichangleo], see if you want to follow Andrew's suggestion and update
BlockStateChange to use slf4j.
I'd be +1 either way - slf4j or existing log4j calls with {{isDebugEnabled}}
checks.
BlockStateChange logs are too noisy
---
Key: HDFS-6860
URL: https://issues.apache.org/jira/browse/HDFS-6860
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.5.0
Reporter: Arpit Agarwal
Assignee: chang li
Labels: newbie
Attachments: HDFS6860.patch
Block State Change logs are too noisy at the default INFO level and affect NN
performance on busy clusters.
Most of these state changes can be logged at debug level instead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6951) Saving namespace and restarting NameNode will remove existing encryption zones
[
https://issues.apache.org/jira/browse/HDFS-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-6951:
---
Attachment: HDFS-6951.002.patch
Resubmitting .002 again so that it is the latest attachment in the hopes that
test-patch will pick this up instead of editsStored.
Saving namespace and restarting NameNode will remove existing encryption zones
--
Key: HDFS-6951
URL: https://issues.apache.org/jira/browse/HDFS-6951
Project: Hadoop HDFS
Issue Type: Sub-task
Components: encryption
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Charles Lamb
Attachments: HDFS-6951-prelim.002.patch, HDFS-6951-testrepo.patch,
HDFS-6951.001.patch, HDFS-6951.002.patch, editsStored
Currently, when users save namespace and restart the NameNode, pre-existing
encryption zones will be wiped out.
I could reproduce this on a pseudo-distributed cluster:
* Create an encryption zone
* List encryption zones and verify the newly created zone is present
* Save the namespace
* Kill and restart the NameNode
* List the encryption zones and you'll find the encryption zone is missing
I've attached a test case for {{TestEncryptionZones}} that reproduces this as
well. Removing the saveNamespace call will get the test to pass.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6951) Saving namespace and restarting NameNode will remove existing encryption zones
[
https://issues.apache.org/jira/browse/HDFS-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-6951:
---
Attachment: (was: HDFS-6951.002.patch)
Saving namespace and restarting NameNode will remove existing encryption zones
--
Key: HDFS-6951
URL: https://issues.apache.org/jira/browse/HDFS-6951
Project: Hadoop HDFS
Issue Type: Sub-task
Components: encryption
Affects Versions: 3.0.0
Reporter: Stephen Chu
Assignee: Charles Lamb
Attachments: HDFS-6951-prelim.002.patch, HDFS-6951-testrepo.patch,
HDFS-6951.001.patch, HDFS-6951.002.patch, editsStored
Currently, when users save namespace and restart the NameNode, pre-existing
encryption zones will be wiped out.
I could reproduce this on a pseudo-distributed cluster:
* Create an encryption zone
* List encryption zones and verify the newly created zone is present
* Save the namespace
* Kill and restart the NameNode
* List the encryption zones and you'll find the encryption zone is missing
I've attached a test case for {{TestEncryptionZones}} that reproduces this as
well. Removing the saveNamespace call will get the test to pass.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Thomas updated HDFS-6800:
---
Attachment: HDFS-6800.8.patch
Good call. Updated patch accordingly.
Determine how Datanode layout changes should interact with rolling upgrade
--
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.8.patch,
HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-6826:
-
Attachment: HDFS-6826v7.6.patch
Uploading v7.6, which is based on v7.4 (based on Jitendra's preference).
This patch adds the {{isClientOp()}} method to the provider which indicates if
the current invocation is done as part of a client FS operation or as part of a
NameNode operation (fs image save, fs edit log, etc). I've had to create an
interceptor for the {{ClientProtocol}}.
The testcases do positive and negative testing for this.
This addresses my action item #1 from my discussion with @daryn.
Plugin interface to enable delegation of HDFS authorization assertions
--
Key: HDFS-6826
URL: https://issues.apache.org/jira/browse/HDFS-6826
Project: Hadoop HDFS
Issue Type: New Feature
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch,
HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch,
HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.6.patch,
HDFS-6826v7.patch, HDFS-6826v8.patch,
HDFSPluggableAuthorizationProposal-v2.pdf,
HDFSPluggableAuthorizationProposal.pdf
When Hbase data, HiveMetaStore data or Search data is accessed via services
(Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
permissions on corresponding entities (databases, tables, views, columns,
search collections, documents). It is desirable, when the data is accessed
directly by users accessing the underlying data files (i.e. from a MapReduce
job), that the permission of the data files map to the permissions of the
corresponding data entity (i.e. table, column family or search collection).
To enable this we need to have the necessary hooks in place in the NameNode
to delegate authorization to an external system that can map HDFS
files/directories to data entities and resolve their permissions based on the
data entities permissions.
I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114347#comment-14114347
]
Arpit Agarwal commented on HDFS-6800:
-
Thanks for all the patch iterations and your patience James. Let me think about
this some more today but I think your latest patch is ready to commit today.
Do you guys have any rolling upgrade testing planned with HDFS-6482 and
HDFS-6800?
Determine how Datanode layout changes should interact with rolling upgrade
--
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.8.patch,
HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6968) Configurable replication degree in the hadoop archive command
[
https://issues.apache.org/jira/browse/HDFS-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-6968:
Attachment: HDFS-6968.patch
Configurable replication degree in the hadoop archive command
-
Key: HDFS-6968
URL: https://issues.apache.org/jira/browse/HDFS-6968
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-6968.patch
Due to be below hard-coded replication degree in {{HadoopArchives}}, the
{{archive}} command will fail if HDFS maximum replication has already been
configured to a number lower than 10.
{code:java}
//increase the replication of src files
jobfs.setReplication(srcFiles, (short) 10);
{code}
This Jira will make the {{archive}} command configurable with desired
replication degree.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6968) Configurable replication degree in the hadoop archive command
[
https://issues.apache.org/jira/browse/HDFS-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-6968:
Status: Patch Available (was: Open)
Configurable replication degree in the hadoop archive command
-
Key: HDFS-6968
URL: https://issues.apache.org/jira/browse/HDFS-6968
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-6968.patch
Due to be below hard-coded replication degree in {{HadoopArchives}}, the
{{archive}} command will fail if HDFS maximum replication has already been
configured to a number lower than 10.
{code:java}
//increase the replication of src files
jobfs.setReplication(srcFiles, (short) 10);
{code}
This Jira will make the {{archive}} command configurable with desired
replication degree.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114352#comment-14114352
]
Arpit Agarwal commented on HDFS-6800:
-
The reason I asked about the testing is so we can start thinking about getting
the changes to branch-2.
Determine how Datanode layout changes should interact with rolling upgrade
--
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.8.patch,
HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114356#comment-14114356
]
James Thomas commented on HDFS-6800:
Great, thanks for your help on this, Arpit. I will try to run some tests today
or tomorrow.
Determine how Datanode layout changes should interact with rolling upgrade
--
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.8.patch,
HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6840) Clients are always sent to the same datanode when read is off rack
[ https://issues.apache.org/jira/browse/HDFS-6840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6840: -- Attachment: hdfs-6840.002.patch Nice catch Yi, new patch that removes that param from the XML file. Clients are always sent to the same datanode when read is off rack -- Key: HDFS-6840 URL: https://issues.apache.org/jira/browse/HDFS-6840 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Andrew Wang Priority: Critical Attachments: hdfs-6840.001.patch, hdfs-6840.002.patch After HDFS-6268 the sorting order of block locations is deterministic for a given block and locality level (e.g.: local, rack. off-rack), so off-rack clients all see the same datanode for the same block. This leads to very poor behavior in distributed cache localization and other scenarios where many clients all want the same block data at approximately the same time. The one datanode is crushed by the load while the other replicas only handle local and rack-local requests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6928) 'hdfs put' command should accept lazyPersist flag for testing
[
https://issues.apache.org/jira/browse/HDFS-6928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-6928:
Attachment: HDFS-6928.03.patch
Thanks for the review Xiaoyu!
Fixed the comment. Also fixed {{TestCLI}} and added two test cases for
{{TestHDFSCLI}}.
'hdfs put' command should accept lazyPersist flag for testing
-
Key: HDFS-6928
URL: https://issues.apache.org/jira/browse/HDFS-6928
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: HDFS-6581
Reporter: Tassapol Athiapinya
Assignee: Arpit Agarwal
Attachments: HDFS-6928.01.patch, HDFS-6928.02.patch,
HDFS-6928.03.patch
Add a '-l' flag to 'hdfs put' which creates the file with the LAZY_PERSIST
option.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114385#comment-14114385
]
Alejandro Abdelnur commented on HDFS-6826:
--
Regarding my action item #2, based on @daryn’s suggestion, I’ve move all data
authz interception to the {{FsPermissionChecker}}, something like:
*Adding the following methods:*
{code}
private String getUserName(INode[] nodes) {
return AuthorizationProvider.get().getUser(nodes);
}
private String getGroupName(INode[] nodes) {
// wee need to recreate INode[] using the snapshot version of the nodes,
here or before calling
return AuthorizationProvider.get().getGroup(nodes);
}
private FsPermission getFsPermission(INode[] nodes) {
return AuthorizationProvider.get().getFsPermission(nodes);
}
private AclFeature getAclFeature(INode[] nodes) {
return AuthorizationProvider.get().getAclFeature(nodes);
}
{code}
And then replacing all the calls user/group/permissions/acls getters of INodes
calls within {{FsPermissionChecker}} to use the above methods, ie:
The intention was to be able to reuse the already calculated {{INode[]}} chain.
The issues I’m running with this are:
* The {{INode[]}} chain calculated on the entry point of checkPermission does
not take into account snapshots, thus we will need to recalculate the
{{INode[]}} to use the right snapshot for the full chain.
* All the logic in {{FSPermissionChecker}} has to be redone to pass the
{{INode[]}} chain around. This will get tricky as in many places direct array
access like this 'checkStickyBit(inodes[inodes.length - 2], last, snapshotId);'
are being done. And this is not always done on the snapshot version of the
{{INode}}.
* The {{INode[]}} may have null elements, complicating things on the plugin
side.
* We’ll have to do the same in the {{FSDirectory}} to create file status.
Another issue is that doing this, the plugin is only intercepting getter calls,
not setter calls.
Overall, it seems the plugin will have to be more complex than with the v7
approach, it will have less functionality (no writes, no snapshots), and it
will require some serious rewriting of the {{FsPermissionChecker}}.
Plugin interface to enable delegation of HDFS authorization assertions
--
Key: HDFS-6826
URL: https://issues.apache.org/jira/browse/HDFS-6826
Project: Hadoop HDFS
Issue Type: New Feature
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
HDFS-6826v3.patch, HDFS-6826v4.patch, HDFS-6826v5.patch, HDFS-6826v6.patch,
HDFS-6826v7.1.patch, HDFS-6826v7.2.patch, HDFS-6826v7.3.patch,
HDFS-6826v7.4.patch, HDFS-6826v7.5.patch, HDFS-6826v7.6.patch,
HDFS-6826v7.patch, HDFS-6826v8.patch,
HDFSPluggableAuthorizationProposal-v2.pdf,
HDFSPluggableAuthorizationProposal.pdf
When Hbase data, HiveMetaStore data or Search data is accessed via services
(Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
permissions on corresponding entities (databases, tables, views, columns,
search collections, documents). It is desirable, when the data is accessed
directly by users accessing the underlying data files (i.e. from a MapReduce
job), that the permission of the data files map to the permissions of the
corresponding data entity (i.e. table, column family or search collection).
To enable this we need to have the necessary hooks in place in the NameNode
to delegate authorization to an external system that can map HDFS
files/directories to data entities and resolve their permissions based on the
data entities permissions.
I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6968) Configurable replication degree in the hadoop archive command
[
https://issues.apache.org/jira/browse/HDFS-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114395#comment-14114395
]
Hadoop QA commented on HDFS-6968:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665075/HDFS-6968.patch
against trunk revision c4c9a78.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-archives.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7838//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7838//console
This message is automatically generated.
Configurable replication degree in the hadoop archive command
-
Key: HDFS-6968
URL: https://issues.apache.org/jira/browse/HDFS-6968
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-6968.patch
Due to be below hard-coded replication degree in {{HadoopArchives}}, the
{{archive}} command will fail if HDFS maximum replication has already been
configured to a number lower than 10.
{code:java}
//increase the replication of src files
jobfs.setReplication(srcFiles, (short) 10);
{code}
This Jira will make the {{archive}} command configurable with desired
replication degree.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6968) Configurable replication degree in the hadoop archive command
[
https://issues.apache.org/jira/browse/HDFS-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114401#comment-14114401
]
Zhe Zhang commented on HDFS-6968:
-
[~andrew.wang] Could you review this? Thx..
Configurable replication degree in the hadoop archive command
-
Key: HDFS-6968
URL: https://issues.apache.org/jira/browse/HDFS-6968
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-6968.patch
Due to be below hard-coded replication degree in {{HadoopArchives}}, the
{{archive}} command will fail if HDFS maximum replication has already been
configured to a number lower than 10.
{code:java}
//increase the replication of src files
jobfs.setReplication(srcFiles, (short) 10);
{code}
This Jira will make the {{archive}} command configurable with desired
replication degree.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6960) Bugfix in LazyWriter, fix test case and some refactoring
[
https://issues.apache.org/jira/browse/HDFS-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114451#comment-14114451
]
Xiaoyu Yao commented on HDFS-6960:
--
In FsDataSetImpl.java Can you put the following into debug log?
{code}
saveNextReplica()
{...
LOG.info(LazyWriter has no blocks to persist. +
Thread going to sleep.);
{code}
Bugfix in LazyWriter, fix test case and some refactoring
Key: HDFS-6960
URL: https://issues.apache.org/jira/browse/HDFS-6960
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, test
Affects Versions: HDFS-6581
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Attachments: HDFS-6960.01.patch
LazyWriter has a bug. While saving the replica to disk we would save it under
{{current/lazyPersist/}}. Instead it should be saved under the appropriate
subdirectory e.g. {{current/lazyPersist/subdir1/subdir0/}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Fix Version/s: (was: 3.0.0) 2.6.0 merged to branch-2. Transparent data at rest encryption --- Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 3.0.0, 2.3.0 Reporter: Alejandro Abdelnur Assignee: Charles Lamb Fix For: 2.6.0 Attachments: HDFS-6134.001.patch, HDFS-6134.002.patch, HDFS-6134_test_plan.pdf, HDFSDataatRestEncryption.pdf, HDFSDataatRestEncryptionProposal_obsolete.pdf, HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf, fs-encryption.2014-08-18.patch, fs-encryption.2014-08-19.patch Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6960) Bugfix in LazyWriter, fix test case and some refactoring
[
https://issues.apache.org/jira/browse/HDFS-6960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114476#comment-14114476
]
Xiaoyu Yao commented on HDFS-6960:
--
The change of return value from void to boolean for discardReplica() is not
being used in the code. Is this necessary?
Bugfix in LazyWriter, fix test case and some refactoring
Key: HDFS-6960
URL: https://issues.apache.org/jira/browse/HDFS-6960
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, test
Affects Versions: HDFS-6581
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Attachments: HDFS-6960.01.patch
LazyWriter has a bug. While saving the replica to disk we would save it under
{{current/lazyPersist/}}. Instead it should be saved under the appropriate
subdirectory e.g. {{current/lazyPersist/subdir1/subdir0/}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
