[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039718#comment-14039718
]
Yongjun Zhang commented on HDFS-6475:
-
Hi,
I uploaded a new patch per [~daryn]'s suggestion. On top of what Daryn
suggested, and because of the exceptions I described in previous update (with
stack information), I added the logic for InvalidToken in ExceptionHandler:
{code}
if (e instanceof SecurityException) {
e = toCause(e);
}
if (e instanceof InvalidToken) {
e = toCause(e);
}
{code}
This logic is essentially what I wanted to share the original getTrueCause
method for.
Hi Daryn, would you please help review again?
BTW, refer to your comment:
{quote}
In saslProcess, just throw the exception instead of running it through
getTrueCause since it's not a "InvalidToken wrapping another exception" anymore.
{quote}
I did what you suggested, but I'm still getting InvalidToken exception (see the
stack described above). So it seems that the exception that saslProcess tries
to handle comes from different source than what I'm running into.
Thanks a lot.
> WebHdfs clients fail without retry because incorrect handling of
> StandbyException
> -
>
> Key: HDFS-6475
> URL: https://issues.apache.org/jira/browse/HDFS-6475
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
> HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
> HDFS-6475.005.patch, HDFS-6475.006.patch, HDFS-6475.007.patch
>
>
> With WebHdfs clients connected to a HA HDFS service, the delegation token is
> previously initialized with the active NN.
> When clients try to issue request, the NN it contacts is stored in a map
> returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
> the NN based on the order, so likely the first one it runs into is StandbyNN.
> If the StandbyNN doesn't have the updated client crediential, it will throw a
> s SecurityException that wraps StandbyException.
> The client is expected to retry another NN, but due to the insufficient
> handling of SecurityException mentioned above, it failed.
> Example message:
> {code}
> {RemoteException={message=Failed to obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken:
> StandbyException, javaCl
> assName=java.lang.SecurityException, exception=SecurityException}}
> org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
> obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
> at kclient1.kclient$1.run(kclient.java:64)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
> at kclient1.kclient.main(kclient.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6527) Edit log corruption due to defered INode removal
[
https://issues.apache.org/jira/browse/HDFS-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039716#comment-14039716
]
Arun C Murthy commented on HDFS-6527:
-
Changed fix version to 2.4.1 since it got merged in for rc1.
> Edit log corruption due to defered INode removal
>
>
> Key: HDFS-6527
> URL: https://issues.apache.org/jira/browse/HDFS-6527
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Fix For: 2.4.1
>
> Attachments: HDFS-6527.branch-2.4.patch, HDFS-6527.trunk.patch,
> HDFS-6527.v2.patch, HDFS-6527.v3.patch, HDFS-6527.v4.patch, HDFS-6527.v5.patch
>
>
> We have seen a SBN crashing with the following error:
> {panel}
> \[Edit log tailer\] ERROR namenode.FSEditLogLoader:
> Encountered exception on operation AddBlockOp
> [path=/xxx,
> penultimateBlock=NULL, lastBlock=blk_111_111, RpcClientId=,
> RpcCallId=-2]
> java.io.FileNotFoundException: File does not exist: /xxx
> {panel}
> This was caused by the deferred removal of deleted inodes from the inode map.
> Since getAdditionalBlock() acquires FSN read lock and then write lock, a
> deletion can happen in between. Because of deferred inode removal outside FSN
> write lock, getAdditionalBlock() can get the deleted inode from the inode map
> with FSN write lock held. This allow addition of a block to a deleted file.
> As a result, the edit log will contain OP_ADD, OP_DELETE, followed by
> OP_ADD_BLOCK. This cannot be replayed by NN, so NN doesn't start up or SBN
> crashes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-6475:
Attachment: HDFS-6475.007.patch
> WebHdfs clients fail without retry because incorrect handling of
> StandbyException
> -
>
> Key: HDFS-6475
> URL: https://issues.apache.org/jira/browse/HDFS-6475
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
> HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
> HDFS-6475.005.patch, HDFS-6475.006.patch, HDFS-6475.007.patch
>
>
> With WebHdfs clients connected to a HA HDFS service, the delegation token is
> previously initialized with the active NN.
> When clients try to issue request, the NN it contacts is stored in a map
> returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
> the NN based on the order, so likely the first one it runs into is StandbyNN.
> If the StandbyNN doesn't have the updated client crediential, it will throw a
> s SecurityException that wraps StandbyException.
> The client is expected to retry another NN, but due to the insufficient
> handling of SecurityException mentioned above, it failed.
> Example message:
> {code}
> {RemoteException={message=Failed to obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken:
> StandbyException, javaCl
> assName=java.lang.SecurityException, exception=SecurityException}}
> org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
> obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
> at kclient1.kclient$1.run(kclient.java:64)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
> at kclient1.kclient.main(kclient.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6527) Edit log corruption due to defered INode removal
[
https://issues.apache.org/jira/browse/HDFS-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-6527:
Fix Version/s: (was: 2.5.0)
2.4.1
> Edit log corruption due to defered INode removal
>
>
> Key: HDFS-6527
> URL: https://issues.apache.org/jira/browse/HDFS-6527
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Blocker
> Fix For: 2.4.1
>
> Attachments: HDFS-6527.branch-2.4.patch, HDFS-6527.trunk.patch,
> HDFS-6527.v2.patch, HDFS-6527.v3.patch, HDFS-6527.v4.patch, HDFS-6527.v5.patch
>
>
> We have seen a SBN crashing with the following error:
> {panel}
> \[Edit log tailer\] ERROR namenode.FSEditLogLoader:
> Encountered exception on operation AddBlockOp
> [path=/xxx,
> penultimateBlock=NULL, lastBlock=blk_111_111, RpcClientId=,
> RpcCallId=-2]
> java.io.FileNotFoundException: File does not exist: /xxx
> {panel}
> This was caused by the deferred removal of deleted inodes from the inode map.
> Since getAdditionalBlock() acquires FSN read lock and then write lock, a
> deletion can happen in between. Because of deferred inode removal outside FSN
> write lock, getAdditionalBlock() can get the deleted inode from the inode map
> with FSN write lock held. This allow addition of a block to a deleted file.
> As a result, the edit log will contain OP_ADD, OP_DELETE, followed by
> OP_ADD_BLOCK. This cannot be replayed by NN, so NN doesn't start up or SBN
> crashes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6585) INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
[
https://issues.apache.org/jira/browse/HDFS-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039694#comment-14039694
]
Hadoop QA commented on HDFS-6585:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651794/patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1262 javac
compiler warnings (more than the trunk's current 1259 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7197//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7197//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7197//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7197//console
This message is automatically generated.
> INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
> ---
>
> Key: HDFS-6585
> URL: https://issues.apache.org/jira/browse/HDFS-6585
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Zhilei Xu
>Assignee: Zhilei Xu
> Labels: patch
> Attachments: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt
>
>
> Most of the APIs (both internal and external) in FSNamesystem calls
> INodesInPath.resolve() to get the list of INodes corresponding to a file
> path. Usually one API will call resolve() multiple times and that's a waste
> of time.
> This issue particularly refers to FSNamesystem.setPermission, which calls
> resolve() twice indirectly: one from checkOwner(), another from
> dir.setPermission().
> Should save the result of resolve(), and use it whenever possible throughout
> the lifetime of an API call, instead of making new resolve() calls.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039675#comment-14039675
]
Yongjun Zhang commented on HDFS-6475:
-
Hello Guys,
Thank you all for the review and discussion.
As a follow-up, the first thing I did was to check that
retriableRetrievePassword does give StandbyException.
Then I tried what [~daryn] suggested how we could fix this jira, and still run
into the same symptom (client side),:
{code}
org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:348)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:108)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:580)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:546)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
at kclient1.kclient1$2.run(kclient1.java:80)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
at kclient1.kclient1.main(kclient1.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
StandbyException
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:108)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
{code}
I traced the server side and found the following:
UserProvider.getValue caught
{{org.apache.hadoop.security.token.SecretManager$InvalidToken:
StandbyException}}
with the following stack
{code}
UserProvider.getValue(HttpContext) line: 56
UserProvider.getValue(HttpContext) line: 41
InjectableValuesProvider.getInjectableValues(HttpContext) line: 46
AbstractResourceMethodDispatchProvider$ResponseOutInvoker(AbstractResourceMethodDispatchProvider$EntityParamInInvoker).getParams(HttpContext)
line: 153
AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(Object,
HttpContext) line: 203
AbstractResourceMethodDispatchProvider$ResponseOutInvoker(ResourceJavaMethodDispatcher).dispatch(Object,
HttpContext) line: 75
HttpMethodRule.accept(CharSequence, Object, UriRuleContext) line: 288
ResourceClassRule.accept(CharSequence, Object, UriRuleContext) line: 108
RightHandPathRule.accept(CharSequence, Object, UriRuleContext) line: 147
RootResourceClassesRule.accept(CharSequence, Object, UriRuleContext) line: 84
WebApplicationImpl._handleRequest(WebApplicationContext, ContainerRequest)
line: 1469
WebApplicationImpl._handleRequest(WebApplicationContext, ContainerRequest,
ContainerResponse) line: 1400
WebApplicationImpl.handleRequest(ContainerRequest, ContainerResponse) line: 1349
WebApplicationImpl.handleRequest(ContainerRequest, ContainerResponseWriter)
line: 1339
ServletContainer$InternalWebComponent(WebComponent).service(URI, URI,
HttpServletRequest, HttpServletResponse) line: 416
ServletContainer.service(URI, URI, HttpServletRequest, HttpServletResponse)
line: 537
ServletContainer.service(HttpServletRequest, HttpServletResponse) line: 699
ServletContainer(HttpServlet).service(ServletRequest, ServletResponse) line:
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039667#comment-14039667 ] Binglin Chang commented on HDFS-4667: - bq. Binglin Chang, do you have any other comments? No, thanks for the patch, lgtm > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, > HDFS-4667.v1.patch, getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jsvc
[
https://issues.apache.org/jira/browse/HDFS-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039649#comment-14039649
]
Hadoop QA commented on HDFS-2856:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651766/HDFS-2856.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7196//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7196//console
This message is automatically generated.
> Fix block protocol so that Datanodes don't require root or jsvc
> ---
>
> Key: HDFS-2856
> URL: https://issues.apache.org/jira/browse/HDFS-2856
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Owen O'Malley
>Assignee: Chris Nauroth
> Attachments: Datanode-Security-Design.pdf,
> Datanode-Security-Design.pdf, Datanode-Security-Design.pdf,
> HDFS-2856.1.patch, HDFS-2856.2.patch, HDFS-2856.3.patch,
> HDFS-2856.prototype.patch
>
>
> Since we send the block tokens unencrypted to the datanode, we currently
> start the datanode as root using jsvc and get a secure (< 1024) port.
> If we have the datanode generate a nonce and send it on the connection and
> the sends an hmac of the nonce back instead of the block token it won't
> reveal any secrets. Thus, we wouldn't require a secure port and would not
> require root or jsvc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6585) INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
[ https://issues.apache.org/jira/browse/HDFS-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6585: Labels: patch (was: ) Status: Patch Available (was: Open) > INodesInPath.resolve is called multiple times in FSNamesystem.setPermission > --- > > Key: HDFS-6585 > URL: https://issues.apache.org/jira/browse/HDFS-6585 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhilei Xu >Assignee: Zhilei Xu > Labels: patch > Attachments: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt > > > Most of the APIs (both internal and external) in FSNamesystem calls > INodesInPath.resolve() to get the list of INodes corresponding to a file > path. Usually one API will call resolve() multiple times and that's a waste > of time. > This issue particularly refers to FSNamesystem.setPermission, which calls > resolve() twice indirectly: one from checkOwner(), another from > dir.setPermission(). > Should save the result of resolve(), and use it whenever possible throughout > the lifetime of an API call, instead of making new resolve() calls. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6585) INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
[ https://issues.apache.org/jira/browse/HDFS-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6585: Attachment: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt > INodesInPath.resolve is called multiple times in FSNamesystem.setPermission > --- > > Key: HDFS-6585 > URL: https://issues.apache.org/jira/browse/HDFS-6585 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhilei Xu >Assignee: Zhilei Xu > Attachments: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt > > > Most of the APIs (both internal and external) in FSNamesystem calls > INodesInPath.resolve() to get the list of INodes corresponding to a file > path. Usually one API will call resolve() multiple times and that's a waste > of time. > This issue particularly refers to FSNamesystem.setPermission, which calls > resolve() twice indirectly: one from checkOwner(), another from > dir.setPermission(). > Should save the result of resolve(), and use it whenever possible throughout > the lifetime of an API call, instead of making new resolve() calls. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6585) INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
Zhilei Xu created HDFS-6585: --- Summary: INodesInPath.resolve is called multiple times in FSNamesystem.setPermission Key: HDFS-6585 URL: https://issues.apache.org/jira/browse/HDFS-6585 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Zhilei Xu Assignee: Zhilei Xu Attachments: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt Most of the APIs (both internal and external) in FSNamesystem calls INodesInPath.resolve() to get the list of INodes corresponding to a file path. Usually one API will call resolve() multiple times and that's a waste of time. This issue particularly refers to FSNamesystem.setPermission, which calls resolve() twice indirectly: one from checkOwner(), another from dir.setPermission(). Should save the result of resolve(), and use it whenever possible throughout the lifetime of an API call, instead of making new resolve() calls. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[
https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039643#comment-14039643
]
Hadoop QA commented on HDFS-6580:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651757/patch_c89bff2bb7a06bb2b0c66a85acbd5113db6b0526.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
org.apache.hadoop.hdfs.TestCrcCorruption
org.apache.hadoop.hdfs.TestFileAppend4
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7195//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7195//console
This message is automatically generated.
> FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
> ---
>
> Key: HDFS-6580
> URL: https://issues.apache.org/jira/browse/HDFS-6580
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Zhilei Xu
>Assignee: Zhilei Xu
> Labels: patch
> Attachments: patch_c89bff2bb7a06bb2b0c66a85acbd5113db6b0526.txt
>
>
> In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file
> info for auditing purpose. getAuditFileInfo() returns null when auditing is
> disabled, and calls dir.getFileInfo() when auditing is enabled.
> One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to
> get file info for auditing. Should change to getAuditFileInfo().
> Note that another internal API, startFileInt() uses dir.getFileInfo()
> correctly, because the returned file stat is returned out.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039624#comment-14039624 ] Hudson commented on HDFS-6222: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5747 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5747/]) HDFS-6222. Remove background token renewer from webhdfs. Contributed by Rushabh Shah and Daryn Sharp. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604300) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenIdentifier.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/SWebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6222.branch-2-v2.patch, > HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039620#comment-14039620
]
Hadoop QA commented on HDFS-6583:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651748/HDFS-6583.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7194//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7194//console
This message is automatically generated.
> Remove clientNode in FileUnderConstructionFeature
> -
>
> Key: HDFS-6583
> URL: https://issues.apache.org/jira/browse/HDFS-6583
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-6583.000.patch, HDFS-6583.001.patch
>
>
> {{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
> {{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
> The reference be recomputed by consulting {{DatanodeManager}}.
> This jira proposes to remove {{clientNode}} in the
> {{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039608#comment-14039608
]
Yi Liu commented on HDFS-6134:
--
{quote}
My point was that we don't necessarily have to know P1 or P2 - if we only know
(P1 xor P2)
{quote}
The pictures have a point, and use an apple with white background. I agree if
we can get information for P2 from (P1 xor P2), then exposing (P1 xor P2) is
not good, just like the pictures.
> Transparent data at rest encryption
> ---
>
> Key: HDFS-6134
> URL: https://issues.apache.org/jira/browse/HDFS-6134
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.3.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFSDataatRestEncryptionProposal_obsolete.pdf,
> HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf
>
>
> Because of privacy and security regulations, for many industries, sensitive
> data at rest must be in encrypted form. For example: the healthÂcare industry
> (HIPAA regulations), the card payment industry (PCI DSS regulations) or the
> US government (FISMA regulations).
> This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
> be used transparently by any application accessing HDFS via Hadoop Filesystem
> Java API, Hadoop libhdfs C library, or WebHDFS REST API.
> The resulting implementation should be able to be used in compliance with
> different regulation requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-6583:
--
Hadoop Flags: Reviewed
+1 the new patch looks good.
> Remove clientNode in FileUnderConstructionFeature
> -
>
> Key: HDFS-6583
> URL: https://issues.apache.org/jira/browse/HDFS-6583
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-6583.000.patch, HDFS-6583.001.patch
>
>
> {{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
> {{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
> The reference be recomputed by consulting {{DatanodeManager}}.
> This jira proposes to remove {{clientNode}} in the
> {{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039592#comment-14039592
]
Hadoop QA commented on HDFS-6583:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651743/HDFS-6583.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7193//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7193//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7193//console
This message is automatically generated.
> Remove clientNode in FileUnderConstructionFeature
> -
>
> Key: HDFS-6583
> URL: https://issues.apache.org/jira/browse/HDFS-6583
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-6583.000.patch, HDFS-6583.001.patch
>
>
> {{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
> {{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
> The reference be recomputed by consulting {{DatanodeManager}}.
> This jira proposes to remove {{clientNode}} in the
> {{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6572) Add an option to the NameNode that prints the software and on-disk image versions
[
https://issues.apache.org/jira/browse/HDFS-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039584#comment-14039584
]
Hadoop QA commented on HDFS-6572:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651735/HDFS-6572.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7192//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7192//console
This message is automatically generated.
> Add an option to the NameNode that prints the software and on-disk image
> versions
> -
>
> Key: HDFS-6572
> URL: https://issues.apache.org/jira/browse/HDFS-6572
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Charles Lamb
>Assignee: Charles Lamb
>Priority: Minor
> Attachments: HDFS-6572.001.patch, HDFS-6572.002.patch
>
>
> The HDFS namenode should have a startup option that prints the metadata
> versions of both the software and the on-disk version. This will be useful
> for debugging certain situations.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039579#comment-14039579 ] Andrew Wang commented on HDFS-6391: --- Gotcha, that works for me. I might rename CipherType to CipherSuite too for parity with TLS, unless Java's Cipher says differently. Thanks for the info Mike! > Get the Key/IV from the NameNode for encrypted files in DFSClient > - > > Key: HDFS-6391 > URL: https://issues.apache.org/jira/browse/HDFS-6391 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Andrew Wang > Attachments: HDFS-6391.1.patch > > > When creating/opening and encrypted file, the DFSClient should get the > encryption key material and the IV for the file in the create/open RPC call. > HDFS admin users would never get key material/IV on encrypted files > create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6222: Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) I have committed this to trunk and branch-2. I co-credited the path to both [~shahrs87] and [~daryn]. Daryn, thank you for the initial patch. Rushabh, thank you for incorporating the code review feedback. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6222.branch-2-v2.patch, > HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039546#comment-14039546 ] Mike Yoder commented on HDFS-6134: -- If you know P1 you can trivially get to P2, of course. My point was that we don't necessarily have to know P1 or P2 - if we only know (P1 xor P2), it's also generally easy to crack - much, much, much less work than AES encryption. Have a look at the wikipedia link above (my source of all knowledge :-) ). > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039540#comment-14039540 ] Mike Yoder commented on HDFS-6391: -- One further detail I forgot to mention - the client cipher list is _ordered_. So the client really does have a lot of control, but the server side has the final say depending on what it can support. > Get the Key/IV from the NameNode for encrypted files in DFSClient > - > > Key: HDFS-6391 > URL: https://issues.apache.org/jira/browse/HDFS-6391 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Andrew Wang > Attachments: HDFS-6391.1.patch > > > When creating/opening and encrypted file, the DFSClient should get the > encryption key material and the IV for the file in the create/open RPC call. > HDFS admin users would never get key material/IV on encrypted files > create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039537#comment-14039537
]
Yi Liu commented on HDFS-6134:
--
[~yoderme], isnât the same as I state?
Also the result
{code}
C1 xor C2 = P1 xor P2
{code}
isnât {code} P2 = C2 xor (C1 xor P1){code} ?
My point is to guess P2, we should know P1. (Absolutely we know C1, C2)
CTR attack is not by finding existing two cipher-text using same encryption key
and same IV, itâs by ability of constructing file and make it encrypted by same
Data key and IV. If we can construct P1, then we know it.
> Transparent data at rest encryption
> ---
>
> Key: HDFS-6134
> URL: https://issues.apache.org/jira/browse/HDFS-6134
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.3.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFSDataatRestEncryptionProposal_obsolete.pdf,
> HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf
>
>
> Because of privacy and security regulations, for many industries, sensitive
> data at rest must be in encrypted form. For example: the healthÂcare industry
> (HIPAA regulations), the card payment industry (PCI DSS regulations) or the
> US government (FISMA regulations).
> This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
> be used transparently by any application accessing HDFS via Hadoop Filesystem
> Java API, Hadoop libhdfs C library, or WebHDFS REST API.
> The resulting implementation should be able to be used in compliance with
> different regulation requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HDFS-6508) Add an XAttr to specify the cipher mode
[ https://issues.apache.org/jira/browse/HDFS-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb resolved HDFS-6508. Resolution: Not a Problem > Add an XAttr to specify the cipher mode > --- > > Key: HDFS-6508 > URL: https://issues.apache.org/jira/browse/HDFS-6508 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security >Reporter: Charles Lamb >Assignee: Charles Lamb > > We should specify the cipher mode in the xattrs for compatibility sake. > Crypto changes over time and we need to prepare for that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039524#comment-14039524 ] Andrew Wang commented on HDFS-6391: --- Ah, so the client would just initially only send the CipherTypes it'd be okay with. I was thinking it was better to let the client choose it's best CipherType (more flexible, since it's per-user rather than global setting), but this works too. The client can also always probe one-by-one if it really wants to choose. > Get the Key/IV from the NameNode for encrypted files in DFSClient > - > > Key: HDFS-6391 > URL: https://issues.apache.org/jira/browse/HDFS-6391 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Andrew Wang > Attachments: HDFS-6391.1.patch > > > When creating/opening and encrypted file, the DFSClient should get the > encryption key material and the IV for the file in the create/open RPC call. > HDFS admin users would never get key material/IV on encrypted files > create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jsvc
[
https://issues.apache.org/jira/browse/HDFS-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-2856:
Attachment: HDFS-2856.3.patch
I'm attaching patch v3 and notifying [~jnp], since I know he had started
reviewing the patch.
The new revision fixes a bug I found in that a client configured with SASL on
DataTransferProtocol would not have been able to communicate with an unsecured
cluster. This is a supported use case for things like distcp from a secured
source to an unsecured destination. The code change to fix this is in
{{SaslDataTransferClient#send}}. It only allows fallback to an unsecure
connection if configuration property
{{ipc.client.fallback-to-simple-auth-allowed}} has been set. This is
consistent with other RPC client code.
I also improved the tests in {{TestSaslDataTransfer}}.
> Fix block protocol so that Datanodes don't require root or jsvc
> ---
>
> Key: HDFS-2856
> URL: https://issues.apache.org/jira/browse/HDFS-2856
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Owen O'Malley
>Assignee: Chris Nauroth
> Attachments: Datanode-Security-Design.pdf,
> Datanode-Security-Design.pdf, Datanode-Security-Design.pdf,
> HDFS-2856.1.patch, HDFS-2856.2.patch, HDFS-2856.3.patch,
> HDFS-2856.prototype.patch
>
>
> Since we send the block tokens unencrypted to the datanode, we currently
> start the datanode as root using jsvc and get a secure (< 1024) port.
> If we have the datanode generate a nonce and send it on the connection and
> the sends an hmac of the nonce back instead of the block token it won't
> reveal any secrets. Thus, we wouldn't require a secure port and would not
> require root or jsvc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jsvc
[ https://issues.apache.org/jira/browse/HDFS-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-2856: Status: Patch Available (was: Open) > Fix block protocol so that Datanodes don't require root or jsvc > --- > > Key: HDFS-2856 > URL: https://issues.apache.org/jira/browse/HDFS-2856 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, security >Affects Versions: 2.4.0, 3.0.0 >Reporter: Owen O'Malley >Assignee: Chris Nauroth > Attachments: Datanode-Security-Design.pdf, > Datanode-Security-Design.pdf, Datanode-Security-Design.pdf, > HDFS-2856.1.patch, HDFS-2856.2.patch, HDFS-2856.3.patch, > HDFS-2856.prototype.patch > > > Since we send the block tokens unencrypted to the datanode, we currently > start the datanode as root using jsvc and get a secure (< 1024) port. > If we have the datanode generate a nonce and send it on the connection and > the sends an hmac of the nonce back instead of the block token it won't > reveal any secrets. Thus, we wouldn't require a secure port and would not > require root or jsvc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jsvc
[ https://issues.apache.org/jira/browse/HDFS-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-2856: Status: Open (was: Patch Available) > Fix block protocol so that Datanodes don't require root or jsvc > --- > > Key: HDFS-2856 > URL: https://issues.apache.org/jira/browse/HDFS-2856 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, security >Affects Versions: 2.4.0, 3.0.0 >Reporter: Owen O'Malley >Assignee: Chris Nauroth > Attachments: Datanode-Security-Design.pdf, > Datanode-Security-Design.pdf, Datanode-Security-Design.pdf, > HDFS-2856.1.patch, HDFS-2856.2.patch, HDFS-2856.prototype.patch > > > Since we send the block tokens unencrypted to the datanode, we currently > start the datanode as root using jsvc and get a secure (< 1024) port. > If we have the datanode generate a nonce and send it on the connection and > the sends an hmac of the nonce back instead of the block token it won't > reveal any secrets. Thus, we wouldn't require a secure port and would not > require root or jsvc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataatRestEncryptionProposal_obsolete.pdf, > HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: (was: ConceptualDesignProposal_2014-06-20.pdf) > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataatRestEncryptionProposal_obsolete.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: (was: ConceptualDesignProposal_2014-06-20.pdf.pdf) > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ConceptualDesignProposal_2014-06-20.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: HDFSDataatRestEncryptionProposal_obsolete.pdf Re-uploading the previous proposal as I've mistakenly deleted it. (using obsolete in the name to avoid confusion). > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ConceptualDesignProposal_2014-06-20.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6134: --- Attachment: ConceptualDesignProposal_2014-06-20.pdf > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ConceptualDesignProposal_2014-06-20.pdf, > HDFSDataatRestEncryptionProposal_obsolete.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: (was: ConceptualDesignProposal_2014-06-19.pdf) > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ConceptualDesignProposal_2014-06-20.pdf.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6134: --- Attachment: ConceptualDesignProposal_2014-06-20.pdf.pdf Revised one-pager. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ConceptualDesignProposal_2014-06-19.pdf, > ConceptualDesignProposal_2014-06-20.pdf.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: (was: HDFSDataAtRestEncryption.pdf) > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ConceptualDesignProposal_2014-06-19.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039491#comment-14039491 ] Jing Zhao commented on HDFS-4667: - Thanks for the review, Nicholas! [~decster], do you have any other comments? > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, > HDFS-4667.v1.patch, getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Description: In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). Note that another internal API, startFileInt() uses dir.getFileInfo() correctly, because the returned file stat is returned out. was: In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). > FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() > --- > > Key: HDFS-6580 > URL: https://issues.apache.org/jira/browse/HDFS-6580 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhilei Xu >Assignee: Zhilei Xu > Labels: patch > Attachments: patch_c89bff2bb7a06bb2b0c66a85acbd5113db6b0526.txt > > > In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file > info for auditing purpose. getAuditFileInfo() returns null when auditing is > disabled, and calls dir.getFileInfo() when auditing is enabled. > One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to > get file info for auditing. Should change to getAuditFileInfo(). > Note that another internal API, startFileInt() uses dir.getFileInfo() > correctly, because the returned file stat is returned out. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Attachment: patch_c89bff2bb7a06bb2b0c66a85acbd5113db6b0526.txt > FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() > --- > > Key: HDFS-6580 > URL: https://issues.apache.org/jira/browse/HDFS-6580 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhilei Xu >Assignee: Zhilei Xu > Labels: patch > Attachments: patch_c89bff2bb7a06bb2b0c66a85acbd5113db6b0526.txt > > > In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file > info for auditing purpose. getAuditFileInfo() returns null when auditing is > disabled, and calls dir.getFileInfo() when auditing is enabled. > One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to > get file info for auditing. Should change to getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5682) Heterogeneous Storage phase 2 - APIs to expose Storage Types
[ https://issues.apache.org/jira/browse/HDFS-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-5682: -- Component/s: namenode datanode Issue Type: New Feature (was: Bug) > Heterogeneous Storage phase 2 - APIs to expose Storage Types > > > Key: HDFS-5682 > URL: https://issues.apache.org/jira/browse/HDFS-5682 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: 20140522-Heterogeneous-Storages-API.pdf > > > Phase 1 (HDFS-2832) added support to present the DataNode as a collection of > discrete storages of different types. > This Jira is to track phase 2 of the Heterogeneous Storage work which > involves exposing Storage Types to applications and adding Quota Management > support for administrators. > This phase will also include tools support for administrators/users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Description: In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). was: In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. 2 internal APIs, startFileInt() and mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). > FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() > --- > > Key: HDFS-6580 > URL: https://issues.apache.org/jira/browse/HDFS-6580 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhilei Xu >Assignee: Zhilei Xu > Labels: patch > > In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file > info for auditing purpose. getAuditFileInfo() returns null when auditing is > disabled, and calls dir.getFileInfo() when auditing is enabled. > One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to > get file info for auditing. Should change to getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6584) Support archival storage
Tsz Wo Nicholas Sze created HDFS-6584: - Summary: Support archival storage Key: HDFS-6584 URL: https://issues.apache.org/jira/browse/HDFS-6584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze In most of the Hadoop clusters, as more and more data is stored for longer time, the demand for storage is outstripping the compute. Hadoop needs a cost effective and easy to manage solution to meet this demand for storage. Current solution is: - Delete the old unused data. This comes at operational cost of identifying unnecessary data and deleting them manually. - Add more nodes to the clusters. This adds along with storage capacity unnecessary compute capacity to the cluster. Hadoop needs a solution to decouple growing storage capacity from compute capacity. Nodes with higher density and less expensive storage with low compute power are becoming available and can be used as cold storage in the clusters. Based on policy the data from hot storage can be moved to cold storage. Adding more nodes to the cold storage can grow the storage independent of the compute capacity in the cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Summary: FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() (was: 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to getAuditFileInfo()) > FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() > --- > > Key: HDFS-6580 > URL: https://issues.apache.org/jira/browse/HDFS-6580 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhilei Xu >Assignee: Zhilei Xu > Labels: patch > > In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file > info for auditing purpose. getAuditFileInfo() returns null when auditing is > disabled, and calls dir.getFileInfo() when auditing is enabled. > 2 internal APIs, startFileInt() and mkdirsInt() mistakenly use the raw > dir.getFileInfo() to get file info for auditing. Should change to > getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Attachment: (was: patch_cadc3030359f6c317e46d43c9f7ef61c5576a872.txt) > FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() > --- > > Key: HDFS-6580 > URL: https://issues.apache.org/jira/browse/HDFS-6580 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhilei Xu >Assignee: Zhilei Xu > Labels: patch > > In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file > info for auditing purpose. getAuditFileInfo() returns null when auditing is > disabled, and calls dir.getFileInfo() when auditing is enabled. > One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to > get file info for auditing. Should change to getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039479#comment-14039479 ] Mike Yoder commented on HDFS-6391: -- Regarding the write-side algo, the way TLS choose the CipherType is that the client sends the list of ciphers it supports, and the server side picks its favorite and returns it. > Get the Key/IV from the NameNode for encrypted files in DFSClient > - > > Key: HDFS-6391 > URL: https://issues.apache.org/jira/browse/HDFS-6391 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Andrew Wang > Attachments: HDFS-6391.1.patch > > > When creating/opening and encrypted file, the DFSClient should get the > encryption key material and the IV for the file in the create/open RPC call. > HDFS admin users would never get key material/IV on encrypted files > create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039474#comment-14039474
]
Jing Zhao commented on HDFS-6562:
-
Comments for the remaining changes:
# I guess we can even put srcParent as a final field in RenameTransaction since
we will not replace the parent INode anymore.
# Can we have a validateOverwrite method to cover all the checks when rename
destination exists (the checks in the "if (dstInode != null)" section)?
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
> HDFS-6562.002.patch, HDFS-6562.003.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039459#comment-14039459 ] Andrew Wang commented on HDFS-6391: --- I'm starting on this JIRA, and was wondering about how we should handle compatibility with regard to cipher types. I'm planning to add a FileEncryptionInfo class, which will incorporate a key, iv, and CipherType enum. Currently, our only CipherType is AES/CTR/NoPadding, but there were some plans in the works to use other CipherTypes as well. The question here relates to CryptoCodec. Right now, the factory is determined based on the client's conf, but for reading an encrypted file, it should be determined by the CipherType of the file. On the read-side, we also need to pass the CipherType from the client to the NN so it's stored correctly. [~hitliuyi], any thoughts on the best way of doing this? From my initial investigation, it looks like the strings used by javax.crypto.Cipher are the standard way of specifying a cipher type. If I provided a mapping from CipherType -> javax.crypto.Cipher-style string, would that be sufficient? I'm also wondering about how the client and server will agree on which cipher to use. The NN needs to interpret the CipherType to know what KeyProvider operations are required. I think it could go something like this on the write-side: * Client specifies the list of CipherTypes it supports * NN filters the list by which it supports * Client chooses a CipherType, or errors out saying "desired CipherType not supported" * Client specifies its chosen CipherType with future commands. Read-side seems pretty trivial: * Client opens a file * NN returns the CipherType for the file * Client either has support for it and reads it, or can't Any thoughts here? > Get the Key/IV from the NameNode for encrypted files in DFSClient > - > > Key: HDFS-6391 > URL: https://issues.apache.org/jira/browse/HDFS-6391 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Andrew Wang > Attachments: HDFS-6391.1.patch > > > When creating/opening and encrypted file, the DFSClient should get the > encryption key material and the IV for the file in the create/open RPC call. > HDFS admin users would never get key material/IV on encrypted files > create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-4667: -- Component/s: (was: datanode) Assignee: Jing Zhao (was: Binglin Chang) Hadoop Flags: Reviewed +1 patch looks good. Thanks Jing. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, > HDFS-4667.v1.patch, getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039444#comment-14039444
]
Jing Zhao edited comment on HDFS-6562 at 6/20/14 10:11 PM:
---
Thanks for the refactoring, Haohui! Just finished reviewing the changes for the
old rename. Some comments:
# Maybe we can let checkSnapshot throw a SnapshotException instead of the
current general IOException. In this way for the old version rename, we do not
need to check the exception msg when validating the rename source.
# Maybe we can rename RenameTransaction to RenameOperation?
# In RenameTransaction, the initialization of srcChildName, isSrcInSnapshot, and
scrChildIsReference can be moved into the constructor so that they can still
be declared as final.
# Since we no longer have INode replacement, we do not need to replace the last
inode of the srcIIP. Simply calling srcChild.recordModification should be good
enough.
{code}
if (isSrcInSnapshot) {
srcChild = srcChild.recordModification(srcIIP.getLatestSnapshotId());
srcIIP.setLastINode(srcChild);
}
{code}
# Similarly I guess we do not need the following code any more (this is in the
new rename):
{code}
if (dstParent.getParent() == null) {
// src and dst file/dir are in the same directory, and the dstParent has
// been replaced when we removed the src. Refresh the dstIIP and
// dstParent.
dstIIP = getINodesInPath4Write(dst, false);
}
{code}
And in removeLastINode(INodesInPath), we can remove the following code:
{code}
INodeDirectory newParent = last.getParent();
if (parent != newParent) {
iip.setINode(-2, newParent);
}
{code}
Will continue reviewing the new rename.
was (Author: jingzhao):
Thanks for the refactoring, Haohui! Just finished reviewing the changes for the
old rename. Some comments:
# Maybe we can let checkSnapshot throw a SnapshotException instead of the
current general IOException. In this way for the old version rename, we do not
need to check the exception msg when validating the rename source.
# Maybe we can rename RenameTransaction to RenameOperation?
# In RenameTransaction, the initialization of srcChildName, isSrcInSnapshot, and
scrChildIsReference can be moved into the constructor so that they can still
be declared as final.
# Since we no longer have INode replacement, we do not need to replace the last
inode of the srcIIP. Simply calling srcChild.recordModification should be good
enough.
{code}
if (isSrcInSnapshot) {
srcChild = srcChild.recordModification(srcIIP.getLatestSnapshotId());
srcIIP.setLastINode(srcChild);
}
{code}
# Similarly I guess we do not need the following code any more:
{code}
if (dstParent.getParent() == null) {
// src and dst file/dir are in the same directory, and the dstParent has
// been replaced when we removed the src. Refresh the dstIIP and
// dstParent.
dstIIP = getINodesInPath4Write(dst, false);
}
{code}
And in removeLastINode(INodesInPath), we can remove the following code:
{code}
INodeDirectory newParent = last.getParent();
if (parent != newParent) {
iip.setINode(-2, newParent);
}
{code}
Will continue reviewing the new rename.
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
> HDFS-6562.002.patch, HDFS-6562.003.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039444#comment-14039444
]
Jing Zhao commented on HDFS-6562:
-
Thanks for the refactoring, Haohui! Just finished reviewing the changes for the
old rename. Some comments:
# Maybe we can let checkSnapshot throw a SnapshotException instead of the
current general IOException. In this way for the old version rename, we do not
need to check the exception msg when validating the rename source.
# Maybe we can rename RenameTransaction to RenameOperation?
# In RenameTransaction, the initialization of srcChildName, isSrcInSnapshot, and
scrChildIsReference can be moved into the constructor so that they can still
be declared as final.
# Since we no longer have INode replacement, we do not need to replace the last
inode of the srcIIP. Simply calling srcChild.recordModification should be good
enough.
{code}
if (isSrcInSnapshot) {
srcChild = srcChild.recordModification(srcIIP.getLatestSnapshotId());
srcIIP.setLastINode(srcChild);
}
{code}
# Similarly I guess we do not need the following code any more:
{code}
if (dstParent.getParent() == null) {
// src and dst file/dir are in the same directory, and the dstParent has
// been replaced when we removed the src. Refresh the dstIIP and
// dstParent.
dstIIP = getINodesInPath4Write(dst, false);
}
{code}
And in removeLastINode(INodesInPath), we can remove the following code:
{code}
INodeDirectory newParent = last.getParent();
if (parent != newParent) {
iip.setINode(-2, newParent);
}
{code}
Will continue reviewing the new rename.
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
> HDFS-6562.002.patch, HDFS-6562.003.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039436#comment-14039436
]
Haohui Mai commented on HDFS-6583:
--
The v1 patch implements [~szetszwo]'s comments.
> Remove clientNode in FileUnderConstructionFeature
> -
>
> Key: HDFS-6583
> URL: https://issues.apache.org/jira/browse/HDFS-6583
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-6583.000.patch, HDFS-6583.001.patch
>
>
> {{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
> {{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
> The reference be recomputed by consulting {{DatanodeManager}}.
> This jira proposes to remove {{clientNode}} in the
> {{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6583:
-
Attachment: HDFS-6583.001.patch
> Remove clientNode in FileUnderConstructionFeature
> -
>
> Key: HDFS-6583
> URL: https://issues.apache.org/jira/browse/HDFS-6583
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-6583.000.patch, HDFS-6583.001.patch
>
>
> {{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
> {{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
> The reference be recomputed by consulting {{DatanodeManager}}.
> This jira proposes to remove {{clientNode}} in the
> {{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039426#comment-14039426
]
Daryn Sharp commented on HDFS-6562:
---
Are you sure this can't be a {{SnapshotAccessControlException}} (which seems
more appropriate) or tunneled out more cleanly?
{noformat}
+} catch (IOException e) {
+ // XXX: The client expects an exception in this case,
+ // and we can't subclass this exception due to backward compatibility
+ String msg = e.getMessage();
+ if (msg.contains(" is snapshottable and already has snapshots")) {
{noformat}
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
> HDFS-6562.002.patch, HDFS-6562.003.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039429#comment-14039429
]
Tsz Wo Nicholas Sze commented on HDFS-6583:
---
This is a good idea. Please as well remove
{code}
final DatanodeDescriptor clientNode =
blockManager.getDatanodeManager().getDatanodeByHost(clientMachine);
{code}
from appendFileInternal and startFileInternal in FSNamesystem.
> Remove clientNode in FileUnderConstructionFeature
> -
>
> Key: HDFS-6583
> URL: https://issues.apache.org/jira/browse/HDFS-6583
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Haohui Mai
> Attachments: HDFS-6583.000.patch
>
>
> {{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
> {{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
> The reference be recomputed by consulting {{DatanodeManager}}.
> This jira proposes to remove {{clientNode}} in the
> {{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-6583:
--
Component/s: namenode
Priority: Minor (was: Major)
Assignee: Haohui Mai
> Remove clientNode in FileUnderConstructionFeature
> -
>
> Key: HDFS-6583
> URL: https://issues.apache.org/jira/browse/HDFS-6583
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
>Priority: Minor
> Attachments: HDFS-6583.000.patch
>
>
> {{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
> {{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
> The reference be recomputed by consulting {{DatanodeManager}}.
> This jira proposes to remove {{clientNode}} in the
> {{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6523) add unit tests for encryption zones api and cli
[ https://issues.apache.org/jira/browse/HDFS-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6523: --- Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6134) > add unit tests for encryption zones api and cli > --- > > Key: HDFS-6523 > URL: https://issues.apache.org/jira/browse/HDFS-6523 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security >Reporter: Charles Lamb >Assignee: Charles Lamb > > Add unit tests for EZ apis and CLI. This work will be done after HDFS-6386, > HDFS-638, and HDFS-6516 are completed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6508) Add an XAttr to specify the cipher mode
[ https://issues.apache.org/jira/browse/HDFS-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6508: --- Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6134) > Add an XAttr to specify the cipher mode > --- > > Key: HDFS-6508 > URL: https://issues.apache.org/jira/browse/HDFS-6508 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security >Reporter: Charles Lamb >Assignee: Charles Lamb > > We should specify the cipher mode in the xattrs for compatibility sake. > Crypto changes over time and we need to prepare for that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039425#comment-14039425
]
Yongjun Zhang commented on HDFS-6475:
-
HI [~daryn],
While I'm working on a revised version, I have a question. Your earlier
suggestion indicated that we should use
{{SecretManager#retriableRetrievePassword}} instead of
{{SecretManager#retrievePassword}}, does that mean client code has to be
modified? If so, is it a compatibility issue? thanks a lot.
> WebHdfs clients fail without retry because incorrect handling of
> StandbyException
> -
>
> Key: HDFS-6475
> URL: https://issues.apache.org/jira/browse/HDFS-6475
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
> HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
> HDFS-6475.005.patch, HDFS-6475.006.patch
>
>
> With WebHdfs clients connected to a HA HDFS service, the delegation token is
> previously initialized with the active NN.
> When clients try to issue request, the NN it contacts is stored in a map
> returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
> the NN based on the order, so likely the first one it runs into is StandbyNN.
> If the StandbyNN doesn't have the updated client crediential, it will throw a
> s SecurityException that wraps StandbyException.
> The client is expected to retry another NN, but due to the insufficient
> handling of SecurityException mentioned above, it failed.
> Example message:
> {code}
> {RemoteException={message=Failed to obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken:
> StandbyException, javaCl
> assName=java.lang.SecurityException, exception=SecurityException}}
> org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
> obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
> at kclient1.kclient$1.run(kclient.java:64)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
> at kclient1.kclient.main(kclient.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6393) User settable xAttr to stop HDFS admins from reading/chowning a file
[ https://issues.apache.org/jira/browse/HDFS-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6393: --- Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6134) > User settable xAttr to stop HDFS admins from reading/chowning a file > > > Key: HDFS-6393 > URL: https://issues.apache.org/jira/browse/HDFS-6393 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > > A user should be able to set an xAttr on any file in HDFS to stop an HDFS > admin user from reading the file. The blacklist for chown/chgrp would also > enforced. > This will stop an HDFS admin from gaining access to job token files and > getting HDFS DelegationTokens that would allow him/her to read an encrypted > file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6390) chown/chgrp users/groups blacklist for encrypted files
[ https://issues.apache.org/jira/browse/HDFS-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6390: --- Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6134) > chown/chgrp users/groups blacklist for encrypted files > --- > > Key: HDFS-6390 > URL: https://issues.apache.org/jira/browse/HDFS-6390 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > > A blacklist of users and groups that stops an admin from changing the > owner/group of the file for encrypted files and directories. This blacklist > would typically contain the regular users used by admins. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6457) Maintain a list of all the Encryption Zones in the file system
[ https://issues.apache.org/jira/browse/HDFS-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6457: --- Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6134) > Maintain a list of all the Encryption Zones in the file system > -- > > Key: HDFS-6457 > URL: https://issues.apache.org/jira/browse/HDFS-6457 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, security >Reporter: Charles Lamb >Assignee: Charles Lamb > > We need to maintain a list of all encryption zones in the file system so that > we can ask questions about what EZ a path belongs to, if any, and let the > admin know all the EZs in the system. > [~andrew.wang] Why not just have a sorted structure with pointers to all the > roots of the EZs? We can populate it during metadata loading on startup, and > keep it updated during runtime. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6555) Specify file encryption attributes at create time
[ https://issues.apache.org/jira/browse/HDFS-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039416#comment-14039416 ] Charles Lamb commented on HDFS-6555: This is important for distcp. > Specify file encryption attributes at create time > - > > Key: HDFS-6555 > URL: https://issues.apache.org/jira/browse/HDFS-6555 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Charles Lamb >Assignee: Charles Lamb > > We need to create a Crypto Blob for passing around crypto info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6555) Specify file encryption attributes at create time
[ https://issues.apache.org/jira/browse/HDFS-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6555: --- Summary: Specify file encryption attributes at create time (was: Create a crypto blob) > Specify file encryption attributes at create time > - > > Key: HDFS-6555 > URL: https://issues.apache.org/jira/browse/HDFS-6555 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Charles Lamb >Assignee: Charles Lamb > > We need to create a Crypto Blob for passing around crypto info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6516) Persist and Restore the List of Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6516: --- Summary: Persist and Restore the List of Encryption Zones (was: Implement List Encryption Zones) > Persist and Restore the List of Encryption Zones > > > Key: HDFS-6516 > URL: https://issues.apache.org/jira/browse/HDFS-6516 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: security >Reporter: Charles Lamb >Assignee: Charles Lamb > Attachments: HDFS-6516.001.patch > > > The list Encryption Zones command (CLI) and backend implementation > (FSNamesystem) needs to be implemented. As part of this, the tests in > TestEncryptionZonesAPI should be updated to use that to validate the results > of the various CreateEZ and DeleteEZ tests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039408#comment-14039408
]
Haohui Mai commented on HDFS-6583:
--
The v0 patch salvages most of the code from HDFS-5714.
> Remove clientNode in FileUnderConstructionFeature
> -
>
> Key: HDFS-6583
> URL: https://issues.apache.org/jira/browse/HDFS-6583
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
> Attachments: HDFS-6583.000.patch
>
>
> {{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
> {{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
> The reference be recomputed by consulting {{DatanodeManager}}.
> This jira proposes to remove {{clientNode}} in the
> {{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6583:
-
Status: Patch Available (was: Open)
> Remove clientNode in FileUnderConstructionFeature
> -
>
> Key: HDFS-6583
> URL: https://issues.apache.org/jira/browse/HDFS-6583
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
> Attachments: HDFS-6583.000.patch
>
>
> {{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
> {{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
> The reference be recomputed by consulting {{DatanodeManager}}.
> This jira proposes to remove {{clientNode}} in the
> {{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6583:
-
Attachment: HDFS-6583.000.patch
> Remove clientNode in FileUnderConstructionFeature
> -
>
> Key: HDFS-6583
> URL: https://issues.apache.org/jira/browse/HDFS-6583
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
> Attachments: HDFS-6583.000.patch
>
>
> {{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
> {{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
> The reference be recomputed by consulting {{DatanodeManager}}.
> This jira proposes to remove {{clientNode}} in the
> {{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
Haohui Mai created HDFS-6583:
Summary: Remove clientNode in FileUnderConstructionFeature
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Attachments: HDFS-6583.000.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Reopened] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HDFS-6391: --- > Get the Key/IV from the NameNode for encrypted files in DFSClient > - > > Key: HDFS-6391 > URL: https://issues.apache.org/jira/browse/HDFS-6391 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Andrew Wang > Attachments: HDFS-6391.1.patch > > > When creating/opening and encrypted file, the DFSClient should get the > encryption key material and the IV for the file in the create/open RPC call. > HDFS admin users would never get key material/IV on encrypted files > create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6572) Add an option to the NameNode that prints the software and on-disk image versions
[ https://issues.apache.org/jira/browse/HDFS-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6572: --- Attachment: HDFS-6572.002.patch New patch with unit test. Previous unit test failure was a spurious timeout. > Add an option to the NameNode that prints the software and on-disk image > versions > - > > Key: HDFS-6572 > URL: https://issues.apache.org/jira/browse/HDFS-6572 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HDFS-6572.001.patch, HDFS-6572.002.patch > > > The HDFS namenode should have a startup option that prints the metadata > versions of both the software and the on-disk version. This will be useful > for debugging certain situations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6582) Missing null check in RpcProgramNfs3#read(XDR, SecurityHandler)
Ted Yu created HDFS-6582:
Summary: Missing null check in RpcProgramNfs3#read(XDR,
SecurityHandler)
Key: HDFS-6582
URL: https://issues.apache.org/jira/browse/HDFS-6582
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Around line 691:
{code}
FSDataInputStream fis = clientCache.getDfsInputStream(userName,
Nfs3Utils.getFileIdPath(handle));
try {
readCount = fis.read(offset, readbuffer, 0, count);
{code}
fis may be null, leading to NullPointerException
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6580) 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to getAuditFileInfo()
[
https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039376#comment-14039376
]
Hadoop QA commented on HDFS-6580:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651704/patch_cadc3030359f6c317e46d43c9f7ef61c5576a872.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
org.apache.hadoop.hdfs.TestLease
org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot
org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
org.apache.hadoop.hdfs.TestDFSClientRetries
org.apache.hadoop.hdfs.server.namenode.TestFsck
org.apache.hadoop.hdfs.server.namenode.TestAuditLogs
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
org.apache.hadoop.hdfs.TestFileAppend4
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
org.apache.hadoop.hdfs.TestReplication
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7190//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7190//console
This message is automatically generated.
> 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to
> getAuditFileInfo()
> ---
>
> Key: HDFS-6580
> URL: https://issues.apache.org/jira/browse/HDFS-6580
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Zhilei Xu
>Assignee: Zhilei Xu
> Labels: patch
> Attachments: patch_cadc3030359f6c317e46d43c9f7ef61c5576a872.txt
>
>
> In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file
> info for auditing purpose. getAuditFileInfo() returns null when auditing is
> disabled, and calls dir.getFileInfo() when auditing is enabled.
> 2 internal APIs, startFileInt() and mkdirsInt() mistakenly use the raw
> dir.getFileInfo() to get file info for auditing. Should change to
> getAuditFileInfo().
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039361#comment-14039361
]
Yongjun Zhang commented on HDFS-6475:
-
Many thanks [~daryn], I will work on a new revision per your suggestion.
> WebHdfs clients fail without retry because incorrect handling of
> StandbyException
> -
>
> Key: HDFS-6475
> URL: https://issues.apache.org/jira/browse/HDFS-6475
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
> HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
> HDFS-6475.005.patch, HDFS-6475.006.patch
>
>
> With WebHdfs clients connected to a HA HDFS service, the delegation token is
> previously initialized with the active NN.
> When clients try to issue request, the NN it contacts is stored in a map
> returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
> the NN based on the order, so likely the first one it runs into is StandbyNN.
> If the StandbyNN doesn't have the updated client crediential, it will throw a
> s SecurityException that wraps StandbyException.
> The client is expected to retry another NN, but due to the insufficient
> handling of SecurityException mentioned above, it failed.
> Example message:
> {code}
> {RemoteException={message=Failed to obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken:
> StandbyException, javaCl
> assName=java.lang.SecurityException, exception=SecurityException}}
> org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
> obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
> at kclient1.kclient$1.run(kclient.java:64)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
> at kclient1.kclient.main(kclient.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039330#comment-14039330
]
Daryn Sharp commented on HDFS-6475:
---
I'm +0 and defer to atm if the fix is harder than what I just suggested.
> WebHdfs clients fail without retry because incorrect handling of
> StandbyException
> -
>
> Key: HDFS-6475
> URL: https://issues.apache.org/jira/browse/HDFS-6475
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
> HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
> HDFS-6475.005.patch, HDFS-6475.006.patch
>
>
> With WebHdfs clients connected to a HA HDFS service, the delegation token is
> previously initialized with the active NN.
> When clients try to issue request, the NN it contacts is stored in a map
> returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
> the NN based on the order, so likely the first one it runs into is StandbyNN.
> If the StandbyNN doesn't have the updated client crediential, it will throw a
> s SecurityException that wraps StandbyException.
> The client is expected to retry another NN, but due to the insufficient
> handling of SecurityException mentioned above, it failed.
> Example message:
> {code}
> {RemoteException={message=Failed to obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken:
> StandbyException, javaCl
> assName=java.lang.SecurityException, exception=SecurityException}}
> org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
> obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
> at kclient1.kclient$1.run(kclient.java:64)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
> at kclient1.kclient.main(kclient.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039327#comment-14039327
]
Daryn Sharp commented on HDFS-6475:
---
What I'm saying is I think the patch adds too much unnecessary code. Filing an
improvement to delete all but a few lines of the code changed in this patch
seems a bit odd. I think you just need to:
# Delete {{getTrueCause}} entirely instead of moving it elsewhere
# In {{saslProcess}}, just throw the exception instead of running it through
{{getTrueCause}} since it's not a "InvalidToken wrapping another exception"
anymore.
# Keep your 3-line change to unwrap {{SecurityException}} in {{toResponse}}
If it turns out to be a lot more complicated, then perhaps a followup jira is
ok, but since what should be the cleanest fix is 4 lines of change it would be
nice to try that first.
> WebHdfs clients fail without retry because incorrect handling of
> StandbyException
> -
>
> Key: HDFS-6475
> URL: https://issues.apache.org/jira/browse/HDFS-6475
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
> HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
> HDFS-6475.005.patch, HDFS-6475.006.patch
>
>
> With WebHdfs clients connected to a HA HDFS service, the delegation token is
> previously initialized with the active NN.
> When clients try to issue request, the NN it contacts is stored in a map
> returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
> the NN based on the order, so likely the first one it runs into is StandbyNN.
> If the StandbyNN doesn't have the updated client crediential, it will throw a
> s SecurityException that wraps StandbyException.
> The client is expected to retry another NN, but due to the insufficient
> handling of SecurityException mentioned above, it failed.
> Example message:
> {code}
> {RemoteException={message=Failed to obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken:
> StandbyException, javaCl
> assName=java.lang.SecurityException, exception=SecurityException}}
> org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
> obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
> at kclient1.kclient$1.run(kclient.java:64)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
> at kclient1.kclient.main(kclient.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[
https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039273#comment-14039273
]
Hadoop QA commented on HDFS-6222:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651716/HDFS-6222.branch-2-v3.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7191//console
This message is automatically generated.
> Remove background token renewer from webhdfs
>
>
> Key: HDFS-6222
> URL: https://issues.apache.org/jira/browse/HDFS-6222
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-6222.branch-2-v2.patch,
> HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch,
> HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch,
> HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch
>
>
> The background token renewer is a source of problems for long-running
> daemons. Webhdfs should lazy fetch a new token when it receives an
> InvalidToken exception.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-6222: - Attachment: HDFS-6222.branch-2-v3.patch Thanks Chris for the review. Please find attached the revised patch for branch-2. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, > HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6222: Hadoop Flags: Reviewed +1 for the patch. Rushabh, thank you for addressing the feedback. Can you please also provide an up-to-date branch-2 patch? I'd be happy to take care of committing this for you. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6579) TestSocketIOWithTimeout#testSocketIOWithTimeout fails on Power PC
[ https://issues.apache.org/jira/browse/HDFS-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6579: Fix Version/s: (was: 2.4.1) > TestSocketIOWithTimeout#testSocketIOWithTimeout fails on Power PC > -- > > Key: HDFS-6579 > URL: https://issues.apache.org/jira/browse/HDFS-6579 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.1.0-beta, 2.0.4-alpha, 2.2.0, 2.3.0, 2.4.0 >Reporter: Jinghui Wang > Attachments: HDFS-6579.patch > > > SocketOutputStream closes its writer if it's partial written. But on PPC, > after writing for some time, buf.capacity still equals buf.remaining. The > reason might be what's written on PPC is buffered,so the buf.remaining will > not change till a flush. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6577) Failure to create FinalizeCommand
[
https://issues.apache.org/jira/browse/HDFS-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039231#comment-14039231
]
Yongjun Zhang commented on HDFS-6577:
-
Thanks a lot Arpit! I will post a patch later today.
> Failure to create FinalizeCommand
> -
>
> Key: HDFS-6577
> URL: https://issues.apache.org/jira/browse/HDFS-6577
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> I visually see a bug in NameNodeRpcServer.java,
> {code}
> @Override // DatanodeProtocol
> public DatanodeCommand blockReport(DatanodeRegistration nodeReg,
> String poolId, StorageBlockReport[] reports) throws IOException {
> verifyRequest(nodeReg);
> if(blockStateChangeLog.isDebugEnabled()) {
> blockStateChangeLog.debug("*BLOCK* NameNode.blockReport: "
>+ "from " + nodeReg + ", reports.length=" + reports.length);
> }
> final BlockManager bm = namesystem.getBlockManager();
> boolean noStaleStorages = false;
> for(StorageBlockReport r : reports) {
> final BlockListAsLongs blocks = new BlockListAsLongs(r.getBlocks());
> noStaleStorages = bm.processReport(nodeReg, r.getStorage(), poolId,
> blocks);
> metrics.incrStorageBlockReportOps();
> }
> if (nn.getFSImage().isUpgradeFinalized() &&
> !nn.isStandbyState() &&
> noStaleStorages) {
> return new FinalizeCommand(poolId);
> }
> return null;
> }
> {code}
> noStaleStorage is initialized to false, then in the for loop {{
> for(StorageBlockReport r : reports) {}} it could be changed to true.
> The problem is, since it's a loop of all reports, the value returned from the
> last blockreport (the last iteration) will set it to the final value. Even if
> it was set to true in the middle of a loop, the final iteration could
> overwrite it to false. Similarly, if it was set to false in the middle, and
> the final iter could overwrite it to true.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6579) TestSocketIOWithTimeout#testSocketIOWithTimeout fails on Power PC
[
https://issues.apache.org/jira/browse/HDFS-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039209#comment-14039209
]
Arpit Agarwal commented on HDFS-6579:
-
Is this ppc64 Linux?
The patch looks fine to me. Just a suggestion, there is OS type detection in
Shell.Java. e.g. see {{Shell.getOSType}}. Perhaps you can extend that to
include your target platform and add a static helper similar to
{{Shell.WINDOWS}}. Not the ideal location to have this logic and there was a
proposal to abstract out platform-specific checks into a separate module but
for now we can at least keep them in one location.
> TestSocketIOWithTimeout#testSocketIOWithTimeout fails on Power PC
> --
>
> Key: HDFS-6579
> URL: https://issues.apache.org/jira/browse/HDFS-6579
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 2.1.0-beta, 2.0.4-alpha, 2.2.0, 2.3.0, 2.4.0
>Reporter: Jinghui Wang
> Fix For: 2.4.1
>
> Attachments: HDFS-6579.patch
>
>
> SocketOutputStream closes its writer if it's partial written. But on PPC,
> after writing for some time, buf.capacity still equals buf.remaining. The
> reason might be what's written on PPC is buffered,so the buf.remaining will
> not change till a flush.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5546) race condition crashes "hadoop ls -R" when directories are moved/removed
[
https://issues.apache.org/jira/browse/HDFS-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039207#comment-14039207
]
Daryn Sharp commented on HDFS-5546:
---
In your example, what are you listing? "hadoop fs -ls -R /" or "hadoop fs -ls
-R /test /other"? Why will it stop? The exception will be caught, displayed,
and it moves to the next item.
BTW, checked gnu coreutils and ls will immediately emit a warning about a path
that disappears, and eventually exit with status 1 when done which matches
FsShell's behavior.
{noformat}
/* Exit statuses. */
enum
{
/* "ls" had a minor problem. E.g., while processing a directory,
ls obtained the name of an entry via readdir, yet was later
unable to stat that name. This happens when listing a directory
in which entries are actively being removed or renamed. */
LS_MINOR_PROBLEM = 1,
{noformat}
> race condition crashes "hadoop ls -R" when directories are moved/removed
>
>
> Key: HDFS-5546
> URL: https://issues.apache.org/jira/browse/HDFS-5546
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-5546.1.patch, HDFS-5546.2.000.patch,
> HDFS-5546.2.001.patch, HDFS-5546.2.002.patch, HDFS-5546.2.003.patch
>
>
> This seems to be a rare race condition where we have a sequence of events
> like this:
> 1. org.apache.hadoop.shell.Ls calls DFS#getFileStatus on directory D.
> 2. someone deletes or moves directory D
> 3. org.apache.hadoop.shell.Ls calls PathData#getDirectoryContents(D), which
> calls DFS#listStatus(D). This throws FileNotFoundException.
> 4. ls command terminates with FNF
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6557) Move the reference of fsimage to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039201#comment-14039201
]
Hudson commented on HDFS-6557:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5745 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5745/])
HDFS-6557. Move the reference of fsimage to FSNamesystem. Contributed by Haohui
Mai. (wheat9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604242)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSPermissionChecker.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
> Move the reference of fsimage to FSNamesystem
> -
>
> Key: HDFS-6557
> URL: https://issues.apache.org/jira/browse/HDFS-6557
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.5.0
>
> Attachments: HDFS-6557.000.patch, HDFS-6557.001.patch
>
>
> Per the suggestion from HDFS-6480. {{FSDirectory}} becomes a in-memory data
> structure so that the reference of fsimage should be moved to
> {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6577) Failure to create FinalizeCommand
[
https://issues.apache.org/jira/browse/HDFS-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039188#comment-14039188
]
Arpit Agarwal commented on HDFS-6577:
-
Sounds good. I can review and commit HDFS-6578 when you have a patch.
> Failure to create FinalizeCommand
> -
>
> Key: HDFS-6577
> URL: https://issues.apache.org/jira/browse/HDFS-6577
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> I visually see a bug in NameNodeRpcServer.java,
> {code}
> @Override // DatanodeProtocol
> public DatanodeCommand blockReport(DatanodeRegistration nodeReg,
> String poolId, StorageBlockReport[] reports) throws IOException {
> verifyRequest(nodeReg);
> if(blockStateChangeLog.isDebugEnabled()) {
> blockStateChangeLog.debug("*BLOCK* NameNode.blockReport: "
>+ "from " + nodeReg + ", reports.length=" + reports.length);
> }
> final BlockManager bm = namesystem.getBlockManager();
> boolean noStaleStorages = false;
> for(StorageBlockReport r : reports) {
> final BlockListAsLongs blocks = new BlockListAsLongs(r.getBlocks());
> noStaleStorages = bm.processReport(nodeReg, r.getStorage(), poolId,
> blocks);
> metrics.incrStorageBlockReportOps();
> }
> if (nn.getFSImage().isUpgradeFinalized() &&
> !nn.isStandbyState() &&
> noStaleStorages) {
> return new FinalizeCommand(poolId);
> }
> return null;
> }
> {code}
> noStaleStorage is initialized to false, then in the for loop {{
> for(StorageBlockReport r : reports) {}} it could be changed to true.
> The problem is, since it's a loop of all reports, the value returned from the
> last blockreport (the last iteration) will set it to the final value. Even if
> it was set to true in the middle of a loop, the final iteration could
> overwrite it to false. Similarly, if it was set to false in the middle, and
> the final iter could overwrite it to true.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6581) Write to single replica in memory
Arpit Agarwal created HDFS-6581: --- Summary: Write to single replica in memory Key: HDFS-6581 URL: https://issues.apache.org/jira/browse/HDFS-6581 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Arpit Agarwal Assignee: Arpit Agarwal Per discussion with the community on HDFS-5851, we will implement writing to a single replica in DN memory via DataTransferProtocol. This avoids some of the issues with short-circuit writes, which we can revisit at a later time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6581 started by Arpit Agarwal. > Write to single replica in memory > - > > Key: HDFS-6581 > URL: https://issues.apache.org/jira/browse/HDFS-6581 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > > Per discussion with the community on HDFS-5851, we will implement writing to > a single replica in DN memory via DataTransferProtocol. > This avoids some of the issues with short-circuit writes, which we can > revisit at a later time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6557) Move the reference of fsimage to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6557:
-
Resolution: Fixed
Fix Version/s: 2.5.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed the patch to trunk and branch-2. Thanks [~jingzhao] for the
review.
> Move the reference of fsimage to FSNamesystem
> -
>
> Key: HDFS-6557
> URL: https://issues.apache.org/jira/browse/HDFS-6557
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.5.0
>
> Attachments: HDFS-6557.000.patch, HDFS-6557.001.patch
>
>
> Per the suggestion from HDFS-6480. {{FSDirectory}} becomes a in-memory data
> structure so that the reference of fsimage should be moved to
> {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-5851) Support memory as a storage medium
[ https://issues.apache.org/jira/browse/HDFS-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-5851: Issue Type: New Feature (was: Sub-task) Parent: (was: HDFS-5682) > Support memory as a storage medium > -- > > Key: HDFS-5851 > URL: https://issues.apache.org/jira/browse/HDFS-5851 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode >Affects Versions: 3.0.0 >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal > Attachments: > SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, > SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf, > SupportingMemoryStorageinHDFSPersistentandDiscardableMemory.pdf > > > Memory can be used as a storage medium for smaller/transient files for fast > write throughput. > More information/design will be added later. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6578) add toString method to DatanodeStorage etc for easier debugging
[
https://issues.apache.org/jira/browse/HDFS-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039183#comment-14039183
]
Arpit Agarwal commented on HDFS-6578:
-
bq. It seems to be nice to add a toString() method for DatanodeStorage class,
so we can print out its key info easier while doing debuging.
The StorageID uniquely identifies the storage, but if you want to print out the
state and type that should be fine.
> add toString method to DatanodeStorage etc for easier debugging
> ---
>
> Key: HDFS-6578
> URL: https://issues.apache.org/jira/browse/HDFS-6578
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> It seems to be nice to add a toString() method for DatanodeStorage class, so
> we can print out its key info easier while doing debuging.
> Another thing is, in the end of BlockManager#processReport, there is the
> following message,
> {code}
>blockLog.info("BLOCK* processReport: from storage " +
> storage.getStorageID()
> + " node " + nodeID + ", blocks: " + newReport.getNumberOfBlocks()
> + ", processing time: " + (endTime - startTime) + " msecs");
> return !node.hasStaleStorages();
> {code}
> We could add node.hasStaleStorages() to the log, and possibly replace
> storage.getSorateID() with the suggested storage.toString().
> Any comments? thanks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6557) Move the reference of fsimage to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039177#comment-14039177
]
Jing Zhao commented on HDFS-6557:
-
The patch looks good to me. +1
> Move the reference of fsimage to FSNamesystem
> -
>
> Key: HDFS-6557
> URL: https://issues.apache.org/jira/browse/HDFS-6557
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6557.000.patch, HDFS-6557.001.patch
>
>
> Per the suggestion from HDFS-6480. {{FSDirectory}} becomes a in-memory data
> structure so that the reference of fsimage should be moved to
> {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6577) Failure to create FinalizeCommand
[
https://issues.apache.org/jira/browse/HDFS-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039172#comment-14039172
]
Yongjun Zhang commented on HDFS-6577:
-
BTW, as part of HDFS-6578 fix, I will add a comment in {{blockReport}} method
to state that the last call is a collective result of all calls to
{{bm.processReport}} in the loop, so other readers won't get confused like I
did:-)
> Failure to create FinalizeCommand
> -
>
> Key: HDFS-6577
> URL: https://issues.apache.org/jira/browse/HDFS-6577
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> I visually see a bug in NameNodeRpcServer.java,
> {code}
> @Override // DatanodeProtocol
> public DatanodeCommand blockReport(DatanodeRegistration nodeReg,
> String poolId, StorageBlockReport[] reports) throws IOException {
> verifyRequest(nodeReg);
> if(blockStateChangeLog.isDebugEnabled()) {
> blockStateChangeLog.debug("*BLOCK* NameNode.blockReport: "
>+ "from " + nodeReg + ", reports.length=" + reports.length);
> }
> final BlockManager bm = namesystem.getBlockManager();
> boolean noStaleStorages = false;
> for(StorageBlockReport r : reports) {
> final BlockListAsLongs blocks = new BlockListAsLongs(r.getBlocks());
> noStaleStorages = bm.processReport(nodeReg, r.getStorage(), poolId,
> blocks);
> metrics.incrStorageBlockReportOps();
> }
> if (nn.getFSImage().isUpgradeFinalized() &&
> !nn.isStandbyState() &&
> noStaleStorages) {
> return new FinalizeCommand(poolId);
> }
> return null;
> }
> {code}
> noStaleStorage is initialized to false, then in the for loop {{
> for(StorageBlockReport r : reports) {}} it could be changed to true.
> The problem is, since it's a loop of all reports, the value returned from the
> last blockreport (the last iteration) will set it to the final value. Even if
> it was set to true in the middle of a loop, the final iteration could
> overwrite it to false. Similarly, if it was set to false in the middle, and
> the final iter could overwrite it to true.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6535) HDFS quota update is wrong when file is appended
[ https://issues.apache.org/jira/browse/HDFS-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039158#comment-14039158 ] Hudson commented on HDFS-6535: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5743 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5743/]) HDFS-6535. HDFS quota update is wrong when file is appended. Contributed by George Wong. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604226) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java > HDFS quota update is wrong when file is appended > > > Key: HDFS-6535 > URL: https://issues.apache.org/jira/browse/HDFS-6535 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: George Wong >Assignee: George Wong > Fix For: 2.5.0 > > Attachments: HDFS-6535.patch, HDFS-6535_v1.patch, TestHDFSQuota.java > > > when a file in the directory with Quota feature is appended, the cached disk > consumption should be updated. > But currently, the update is wrong. > Use the uploaded UT to reproduce this bug. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039156#comment-14039156
]
Yongjun Zhang commented on HDFS-6475:
-
HI [~daryn],
Thanks a lot for the review and the in-depth comments. Since the
{{getTrueCause}} method is already there, and the patch I posted reused it to
solve the problem. What you suggested would be an improvement of the solution,
would you please consider that we commit this patch first, and file a follow-up
jira to improve the solution? Really appreciate your help.
> WebHdfs clients fail without retry because incorrect handling of
> StandbyException
> -
>
> Key: HDFS-6475
> URL: https://issues.apache.org/jira/browse/HDFS-6475
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
> HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
> HDFS-6475.005.patch, HDFS-6475.006.patch
>
>
> With WebHdfs clients connected to a HA HDFS service, the delegation token is
> previously initialized with the active NN.
> When clients try to issue request, the NN it contacts is stored in a map
> returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
> the NN based on the order, so likely the first one it runs into is StandbyNN.
> If the StandbyNN doesn't have the updated client crediential, it will throw a
> s SecurityException that wraps StandbyException.
> The client is expected to retry another NN, but due to the insufficient
> handling of SecurityException mentioned above, it failed.
> Example message:
> {code}
> {RemoteException={message=Failed to obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken:
> StandbyException, javaCl
> assName=java.lang.SecurityException, exception=SecurityException}}
> org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
> obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
> at kclient1.kclient$1.run(kclient.java:64)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
> at kclient1.kclient.main(kclient.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6580) 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6580: Assignee: Zhilei Xu > 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to > getAuditFileInfo() > --- > > Key: HDFS-6580 > URL: https://issues.apache.org/jira/browse/HDFS-6580 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhilei Xu >Assignee: Zhilei Xu > Labels: patch > Attachments: patch_cadc3030359f6c317e46d43c9f7ef61c5576a872.txt > > > In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file > info for auditing purpose. getAuditFileInfo() returns null when auditing is > disabled, and calls dir.getFileInfo() when auditing is enabled. > 2 internal APIs, startFileInt() and mkdirsInt() mistakenly use the raw > dir.getFileInfo() to get file info for auditing. Should change to > getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Attachment: patch_cadc3030359f6c317e46d43c9f7ef61c5576a872.txt > 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to > getAuditFileInfo() > --- > > Key: HDFS-6580 > URL: https://issues.apache.org/jira/browse/HDFS-6580 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhilei Xu > Labels: patch > Attachments: patch_cadc3030359f6c317e46d43c9f7ef61c5576a872.txt > > > In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file > info for auditing purpose. getAuditFileInfo() returns null when auditing is > disabled, and calls dir.getFileInfo() when auditing is enabled. > 2 internal APIs, startFileInt() and mkdirsInt() mistakenly use the raw > dir.getFileInfo() to get file info for auditing. Should change to > getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Status: Patch Available (was: Open) > 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to > getAuditFileInfo() > --- > > Key: HDFS-6580 > URL: https://issues.apache.org/jira/browse/HDFS-6580 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Zhilei Xu > Labels: patch > Attachments: patch_cadc3030359f6c317e46d43c9f7ef61c5576a872.txt > > > In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file > info for auditing purpose. getAuditFileInfo() returns null when auditing is > disabled, and calls dir.getFileInfo() when auditing is enabled. > 2 internal APIs, startFileInt() and mkdirsInt() mistakenly use the raw > dir.getFileInfo() to get file info for auditing. Should change to > getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6580) 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to getAuditFileInfo()
Zhilei Xu created HDFS-6580: --- Summary: 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to getAuditFileInfo() Key: HDFS-6580 URL: https://issues.apache.org/jira/browse/HDFS-6580 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Zhilei Xu In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. 2 internal APIs, startFileInt() and mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6535) HDFS quota update is wrong when file is appended
[ https://issues.apache.org/jira/browse/HDFS-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6535: Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk and branch-2. Thanks for the contribution, [~GWong]! > HDFS quota update is wrong when file is appended > > > Key: HDFS-6535 > URL: https://issues.apache.org/jira/browse/HDFS-6535 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: George Wong >Assignee: George Wong > Fix For: 2.5.0 > > Attachments: HDFS-6535.patch, HDFS-6535_v1.patch, TestHDFSQuota.java > > > when a file in the directory with Quota feature is appended, the cached disk > consumption should be updated. > But currently, the update is wrong. > Use the uploaded UT to reproduce this bug. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5546) race condition crashes "hadoop ls -R" when directories are moved/removed
[
https://issues.apache.org/jira/browse/HDFS-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039140#comment-14039140
]
Lei (Eddy) Xu commented on HDFS-5546:
-
My question is that, suppose we have directories {{/test/0..10}} and
{{/other/1..10}}, and deleted {{/test/1}} during execution. The first patch
will only print {{/test/0}} and {{/other/1..10}}, so that {{/test/2..10}} are
ignored in this case.
> race condition crashes "hadoop ls -R" when directories are moved/removed
>
>
> Key: HDFS-5546
> URL: https://issues.apache.org/jira/browse/HDFS-5546
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-5546.1.patch, HDFS-5546.2.000.patch,
> HDFS-5546.2.001.patch, HDFS-5546.2.002.patch, HDFS-5546.2.003.patch
>
>
> This seems to be a rare race condition where we have a sequence of events
> like this:
> 1. org.apache.hadoop.shell.Ls calls DFS#getFileStatus on directory D.
> 2. someone deletes or moves directory D
> 3. org.apache.hadoop.shell.Ls calls PathData#getDirectoryContents(D), which
> calls DFS#listStatus(D). This throws FileNotFoundException.
> 4. ls command terminates with FNF
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6535) HDFS quota update is wrong when file is appended
[ https://issues.apache.org/jira/browse/HDFS-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039134#comment-14039134 ] Jing Zhao commented on HDFS-6535: - The patch looks good to me. +1. I will commit it shortly. > HDFS quota update is wrong when file is appended > > > Key: HDFS-6535 > URL: https://issues.apache.org/jira/browse/HDFS-6535 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.4.0 >Reporter: George Wong >Assignee: George Wong > Attachments: HDFS-6535.patch, HDFS-6535_v1.patch, TestHDFSQuota.java > > > when a file in the directory with Quota feature is appended, the cached disk > consumption should be updated. > But currently, the update is wrong. > Use the uploaded UT to reproduce this bug. -- This message was sent by Atlassian JIRA (v6.2#6252)
