[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317901#comment-15317901
]
Hadoop QA commented on HDFS-10458:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 3m 51s
{color} | {color:red} Docker failed to build yetus/hadoop:babe025. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808568/HDFS-10458-branch-2.00.patch
|
| JIRA Issue | HDFS-10458 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15680/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: HDFS-10458-branch-2.00.patch,
> HDFS-10458-branch-2.6.00.patch, HDFS-10458-branch-2.7.00.patch,
> HDFS-10458.00.patch, HDFS-10458.03.patch, HDFS-10458.04.patch,
> HDFS-10458.05.patch, HDFSA-10458.01.patch, HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317900#comment-15317900
]
Wangda Tan commented on HDFS-10458:
---
Thanks, [~zhz]!
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: HDFS-10458-branch-2.00.patch,
> HDFS-10458-branch-2.6.00.patch, HDFS-10458-branch-2.7.00.patch,
> HDFS-10458.00.patch, HDFS-10458.03.patch, HDFS-10458.04.patch,
> HDFS-10458.05.patch, HDFSA-10458.01.patch, HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10458:
-
Attachment: HDFS-10458-branch-2.00.patch
Thanks Wangda for pointing this out. Yes I believe it is a JDK version issue.
Attaching a branch-2 patch to verify with Jenkins.
I don't think we need to revert it from branch-2.7. The 2.7 patch was rebased,
and the compilation issue ({{Generics}} automatically type matching in
constructor) was resolved at the rebase time.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: HDFS-10458-branch-2.00.patch,
> HDFS-10458-branch-2.6.00.patch, HDFS-10458-branch-2.7.00.patch,
> HDFS-10458.00.patch, HDFS-10458.03.patch, HDFS-10458.04.patch,
> HDFS-10458.05.patch, HDFSA-10458.01.patch, HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9924) [umbrella] Asynchronous HDFS Access
[
https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317887#comment-15317887
]
Jitendra Nath Pandey commented on HDFS-9924:
I think {{AsyncDistributedFileSystem}} should be annotated as Private, similar
to {{DistributedFileSystem}} to further highlight that it is not external
facing.
> [umbrella] Asynchronous HDFS Access
> ---
>
> Key: HDFS-9924
> URL: https://issues.apache.org/jira/browse/HDFS-9924
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: AsyncHdfs20160510.pdf
>
>
> This is an umbrella JIRA for supporting Asynchronous HDFS Access.
> Currently, all the API methods are blocking calls -- the caller is blocked
> until the method returns. It is very slow if a client makes a large number
> of independent calls in a single thread since each call has to wait until the
> previous call is finished. It is inefficient if a client needs to create a
> large number of threads to invoke the calls.
> We propose adding a new API to support asynchronous calls, i.e. the caller is
> not blocked. The methods in the new API immediately return a Java Future
> object. The return value can be obtained by the usual Future.get() method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317884#comment-15317884
]
Hadoop QA commented on HDFS-10458:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
39s {color} | {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 43s
{color} | {color:red} hadoop-hdfs in branch-2.6 failed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 46s
{color} | {color:red} hadoop-hdfs in branch-2.6 failed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s
{color} | {color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 52s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2.6 has 273
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} branch-2.6 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s
{color} | {color:green} branch-2.6 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 52s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 52s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 48s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 48s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3402 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 1m 14s
{color} | {color:red} The patch 74 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 40s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 42s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 30s
{color} | {color:red} The patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 49s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:44eef0e |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808548/HDFS-10458-branch-2.6.00.patch
|
| JIRA Issue |
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317881#comment-15317881
]
Wangda Tan commented on HDFS-10458:
---
[~zhz], perhaps this causes by different JDK version, etc. I didn't investigate
further for that. This is Jenkins run with the patch:
https://builds.apache.org/job/HADOOP2_Release_Artifacts_Builder/72/console.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: HDFS-10458-branch-2.6.00.patch,
> HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317877#comment-15317877
]
Zhe Zhang commented on HDFS-10458:
--
Jenkins run [results |
https://issues.apache.org/jira/browse/HDFS-10458?focusedCommentId=15317653=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15317653]
for branch-2.7 patch.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: HDFS-10458-branch-2.6.00.patch,
> HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317875#comment-15317875
]
Zhe Zhang commented on HDFS-10458:
--
Thanks for the update [~leftnoteasy]. I did build locally before pushing to
branch-2 and branch-2.8. I also did a fresh Jenkins run for branch-2.7 and it
passed.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: HDFS-10458-branch-2.6.00.patch,
> HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317872#comment-15317872
]
Wangda Tan commented on HDFS-10458:
---
branch-2 has the same issue, reverting patch from branch-2 as well.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: HDFS-10458-branch-2.6.00.patch,
> HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317868#comment-15317868
]
Wangda Tan commented on HDFS-10458:
---
[~zhz], I just noticed this breaks branch-2.8 build. So I reverted this from
branch-2.8.
Could you check if it breaks any other branches as well?
Thanks,
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: HDFS-10458-branch-2.6.00.patch,
> HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314624#comment-15314624
]
John Zhuge edited comment on HDFS-6962 at 6/7/16 5:07 AM:
--
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in
Hadoop 2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* Preserve the existing meaning of field {{FsPermissionProto masked}}: mode +
umask.
* Please note this approach is slightly different from what you suggested
above. I am ok either way but find this a little less possibility for
misunderstanding.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
if (child.isPosixAclInheritanceEnabled() == true) {
//
// HDFS-6962. POSIX ACL inheritance.
//
// Set permission to unmasked if necessary
if (!child.getFsPermission().equals(child.getUnmaskedMode())) {
child.setPermission(child.getUnmaskedMode());
}
if (copyINodeDefaultAclInternal(child) == false) {
// No default ACL in parent dir
// Set permission to masked
child.setPermission(child.getMaskedMode());
}
} else {
//
// Old behavior before HDFS-6962.
//
// Set permission to masked if necessary
if (!child.getFsPermission().equals(child.getMaskedMode())) {
child.setPermission(child.getMaskedMode());
}
copyINodeDefaultAclInternal(child);
}
{code}
Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient;
{{INode#getMaskedMode}} returns the masked mode;
{{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir
request comes from a client with HDFS-6962 fix and
{{dfs.namenode.posix.acl.inheritance.enabled}} is true.
was (Author: jzhuge):
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in
Hadoop 2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* Preserve the existing meaning of field {{FsPermissionProto masked}}: mode +
umask.
* Please note this approach is slightly different from what you suggested
above. I am ok either way but find this a little less possibility for
misunderstanding.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
if (child.isPosixAclInheritanceEnabled() == true) {
/*
* HDFS-6962. POSIX ACL inheritance.
*/
// Set permission to unmasked if necessary
if (!child.getFsPermission().equals(child.getUnmaskedMode())) {
child.setPermission(child.getUnmaskedMode());
}
if (AclStorage.copyINodeDefaultAclInternal(child) == false) {
// No default ACL in parent dir
// Set permission to masked
child.setPermission(child.getMaskedMode());
}
} else {
/*
* Old behavior before HDFS-6962.
*/
// Set permission to masked if necessary
if (!child.getFsPermission().equals(child.getMaskedMode())) {
child.setPermission(child.getMaskedMode());
}
AclStorage.copyINodeDefaultAclInternal(child);
}
{code}
Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient;
{{INode#getMaskedMode}} returns the masked mode;
{{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir
request come from a client that supports POSIX ACL inheritance and
{{dfs.namenode.posix.acl.inheritance.enabled}} is true.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
>
[jira] [Commented] (HDFS-9924) [umbrella] Asynchronous HDFS Access
[
https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317858#comment-15317858
]
Jitendra Nath Pandey commented on HDFS-9924:
The features are not played or experimented with if they are in a feature
branch. No one will use if it is not part of a release, or at least in a
release branch or trunk.
The best way to actually harden an API is to have downstream users use them and
provide feedback. @Unstable was defined for exactly the same purpose.
As suggested by [~steve_l] earlier, if it helps an additional annotation
@Experimental can be created to more strongly highlight the
unstable/experimental nature of the API.
The current implementation is not a user facing API therefore it will have to
be modified when final API is decided in HADOOP-12910.
bq. ... figuring a general approach for async'ing the Filesystem.
There is a proposal on HADOOP-12910 and lets have a discussion around that, to
ensure right API gets committed.
bq ... Retrofitting callback on Future,
I agree it is not a good idea. I think CompletableFuture in jdk8 seems like a
good choice. I am leaning to support CompletableFuture in trunk, and if its too
hard to mimic in branch-2, don't expose a user facing API in branch-2 at all.
Users in branch-2 can use the current unstable/experimental version in
{{DistributedFileSystem}} to try out. Since CompletableFuture implements
Future, it will still not be incompatible in trunk, although we don't guarantee
that.
> [umbrella] Asynchronous HDFS Access
> ---
>
> Key: HDFS-9924
> URL: https://issues.apache.org/jira/browse/HDFS-9924
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Xiaobing Zhou
> Attachments: AsyncHdfs20160510.pdf
>
>
> This is an umbrella JIRA for supporting Asynchronous HDFS Access.
> Currently, all the API methods are blocking calls -- the caller is blocked
> until the method returns. It is very slow if a client makes a large number
> of independent calls in a single thread since each call has to wait until the
> previous call is finished. It is inefficient if a client needs to create a
> large number of threads to invoke the calls.
> We propose adding a new API to support asynchronous calls, i.e. the caller is
> not blocked. The methods in the new API immediately return a Java Future
> object. The return value can be obtained by the usual Future.get() method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-6962) ACLs inheritance conflict with umaskmode
[
https://issues.apache.org/jira/browse/HDFS-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314624#comment-15314624
]
John Zhuge edited comment on HDFS-6962 at 6/7/16 5:04 AM:
--
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance.enabled}}, default false in
Hadoop 2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* Preserve the existing meaning of field {{FsPermissionProto masked}}: mode +
umask.
* Please note this approach is slightly different from what you suggested
above. I am ok either way but find this a little less possibility for
misunderstanding.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
if (child.isPosixAclInheritanceEnabled() == true) {
/*
* HDFS-6962. POSIX ACL inheritance.
*/
// Set permission to unmasked if necessary
if (!child.getFsPermission().equals(child.getUnmaskedMode())) {
child.setPermission(child.getUnmaskedMode());
}
if (AclStorage.copyINodeDefaultAclInternal(child) == false) {
// No default ACL in parent dir
// Set permission to masked
child.setPermission(child.getMaskedMode());
}
} else {
/*
* Old behavior before HDFS-6962.
*/
// Set permission to masked if necessary
if (!child.getFsPermission().equals(child.getMaskedMode())) {
child.setPermission(child.getMaskedMode());
}
AclStorage.copyINodeDefaultAclInternal(child);
}
{code}
Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient;
{{INode#getMaskedMode}} returns the masked mode;
{{INode#isPosixAclInheritanceEnabled}} returns true when the create/mkdir
request come from a client that supports POSIX ACL inheritance and
{{dfs.namenode.posix.acl.inheritance.enabled}} is true.
was (Author: jzhuge):
Hi [~cnauroth],
In order to maintain backwards compatibility in Hadoop 2.x, could we add a new
NameNode flag {{dfs.namenode.posix.acl.inheritance}}, default false in Hadoop
2.x, default true in Hadoop 3?
Changes to message {{CreateRequestProto}} and {{MkdirsRequestProto}}
* Add optional field {{FsPermissionProto unmasked}} to store unmasked mode
parameter
* The meaning of the existing field {{FsPermissionProto masked}} stays the
same: mode + umask, for both old and new client.
* Please note this approach is slightly different alternative what you
suggested. I am ok either way.
Wrap around {{AclStorage#copyINodeDefaultAcl}}:
{code:java}
public static void copyINodeDefaultAcl(INode child) {
// Current permission is masked
assert child.getFsPermission().equals(child.getMaskedMode());
if (child.getPosixAclInheritance()) {
// Set permission to unmasked
child.setPermission(child.getUnmaskedMode());
}
if (AclStorage.copyINodeDefaultAclInternal(child) == false) {
// No default ACL in parent dir
if (child.getPosixAclInheritance()) {
// Set permission to masked
child.setPermission(child.getMaskedMode());
}
}
}
{code}
Here {{INode#getUnmaskedMode}} returns the unmasked mode sent by DFSClient;
{{INode#getMaskedMode}} returns the masked mode;
{{INode#getPosixAclInheritance}} returns true when from a new client and the
flag is on.
> ACLs inheritance conflict with umaskmode
>
>
> Key: HDFS-6962
> URL: https://issues.apache.org/jira/browse/HDFS-6962
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Affects Versions: 2.4.1
> Environment: CentOS release 6.5 (Final)
>Reporter: LINTE
>Assignee: John Zhuge
>Priority: Critical
> Labels: hadoop, security
> Attachments: HDFS-6962.001.patch, HDFS-6962.1.patch
>
>
> In hdfs-site.xml
>
> dfs.umaskmode
> 027
>
> 1/ Create a directory as superuser
> bash# hdfs dfs -mkdir /tmp/ACLS
> 2/ set default ACLs on this directory rwx access for group readwrite and user
> toto
> bash# hdfs dfs -setfacl -m default:group:readwrite:rwx /tmp/ACLS
> bash# hdfs dfs -setfacl -m default:user:toto:rwx /tmp/ACLS
> 3/ check ACLs /tmp/ACLS/
> bash# hdfs dfs -getfacl /tmp/ACLS/
> # file: /tmp/ACLS
> # owner: hdfs
> # group: hadoop
> user::rwx
> group::r-x
> other::---
> default:user::rwx
> default:user:toto:rwx
> default:group::r-x
> default:group:readwrite:rwx
> default:mask::rwx
> default:other::---
> user::rwx | group::r-x | other::--- matches with the umaskmode defined in
> hdfs-site.xml, everything ok !
> default:group:readwrite:rwx allow readwrite group with rwx access for
> inhéritance.
> default:user:toto:rwx allow toto user with rwx
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317844#comment-15317844
]
Hadoop QA commented on HDFS-10458:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s {color} | {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 50s
{color} | {color:red} hadoop-hdfs in branch-2.6 failed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 44s
{color} | {color:red} hadoop-hdfs in branch-2.6 failed with JDK v1.7.0_101.
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s
{color} | {color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 1s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in branch-2.6 has 273
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} branch-2.6 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s
{color} | {color:green} branch-2.6 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 48s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 48s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 41s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 41s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 3022 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 1m 2s
{color} | {color:red} The patch 74 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 45s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 46s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 38s
{color} | {color:red} The patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 56s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:44eef0e |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808548/HDFS-10458-branch-2.6.00.patch
|
| JIRA Issue |
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317820#comment-15317820
]
Hadoop QA commented on HDFS-10458:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 3m 41s
{color} | {color:red} Docker failed to build yetus/hadoop:44eef0e. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808548/HDFS-10458-branch-2.6.00.patch
|
| JIRA Issue | HDFS-10458 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15676/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: HDFS-10458-branch-2.6.00.patch,
> HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10458:
-
Fix Version/s: 3.0.0-alpha1
2.7.3
2.8.0
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Fix For: 2.8.0, 2.7.3, 3.0.0-alpha1
>
> Attachments: HDFS-10458-branch-2.6.00.patch,
> HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10458:
-
Attachment: HDFS-10458-branch-2.6.00.patch
All reported branch-2.7 failures are unrelated and pass locally.
Back porting to 2.6 was almost clean, but attaching patch to trigger Jenkins to
be safe.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10458-branch-2.6.00.patch,
> HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10485) Fix findbugs warning in FSEditLog.java
[
https://issues.apache.org/jira/browse/HDFS-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317725#comment-15317725
]
Hadoop QA commented on HDFS-10485:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 0 new + 127 unchanged - 2 fixed = 127 total (was 129) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 35s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestAsyncHDFSWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808525/HDFS-10485.01.patch |
| JIRA Issue | HDFS-10485 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 17612163806b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6de9213 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15673/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15673/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15673/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15673/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fix findbugs warning in FSEditLog.java
>
[jira] [Commented] (HDFS-9924) [umbrella] Asynchronous HDFS Access
[ https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317717#comment-15317717 ] stack commented on HDFS-9924: - Ugh. I meant to say, we risk having different 'async' API/implementations if we piecemeal the implementation ahead of our figuring a general approach for async'ing the Filesystem. What is committed currently is inadequate according to the discussion so far missing callback. Retrofitting callback on Future, as I understand it, will require a different implementation; therefore the commits are premature. Revert in the meantime seems like the right thing to do. bq. I'm much more worried about API correctness Waiting a while to actually let more folks play with it before pushing it into a release (including the 3.x release that we're working to cut from trunk) just seems like an obvious, common sense thing to do. Above makes sense to me. > [umbrella] Asynchronous HDFS Access > --- > > Key: HDFS-9924 > URL: https://issues.apache.org/jira/browse/HDFS-9924 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Reporter: Tsz Wo Nicholas Sze >Assignee: Xiaobing Zhou > Attachments: AsyncHdfs20160510.pdf > > > This is an umbrella JIRA for supporting Asynchronous HDFS Access. > Currently, all the API methods are blocking calls -- the caller is blocked > until the method returns. It is very slow if a client makes a large number > of independent calls in a single thread since each call has to wait until the > previous call is finished. It is inefficient if a client needs to create a > large number of threads to invoke the calls. > We propose adding a new API to support asynchronous calls, i.e. the caller is > not blocked. The methods in the new API immediately return a Java Future > object. The return value can be obtained by the usual Future.get() method. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10440) Improve DataNode web UI
[ https://issues.apache.org/jira/browse/HDFS-10440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-10440: --- Affects Version/s: (was: 2.6.0) (was: 2.5.0) Target Version/s: 2.9.0, 3.0.0-alpha1 (was: 2.8.0) Component/s: ui > Improve DataNode web UI > --- > > Key: HDFS-10440 > URL: https://issues.apache.org/jira/browse/HDFS-10440 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ui >Affects Versions: 2.7.0 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-10440.001.patch, HDFS-10440.002.patch, > datanode_2nns.html.002.jpg, datanode_html.001.jpg, > datanode_loading_err.002.jpg, datanode_utilities.001.jpg, > datanode_utilities.002.jpg, dn_web_ui_mockup.jpg > > > At present, datanode web UI doesn't have much information except for node > name and port. Propose to add more information similar to namenode UI, > including, > * Static info (version, block pool and cluster ID) > * Block pools info (BP IDs, namenode address, actor states) > * Storage info (Volumes, capacity used, reserved, left) > * Utilities (logs) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10440) Improve DataNode web UI
[ https://issues.apache.org/jira/browse/HDFS-10440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317713#comment-15317713 ] Weiwei Yang commented on HDFS-10440: [~kihwal] Yep, that makes sense to me, I just created a new JIRA HDFS-10493 to add links (Like what old UI provides). > Improve DataNode web UI > --- > > Key: HDFS-10440 > URL: https://issues.apache.org/jira/browse/HDFS-10440 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-10440.001.patch, HDFS-10440.002.patch, > datanode_2nns.html.002.jpg, datanode_html.001.jpg, > datanode_loading_err.002.jpg, datanode_utilities.001.jpg, > datanode_utilities.002.jpg, dn_web_ui_mockup.jpg > > > At present, datanode web UI doesn't have much information except for node > name and port. Propose to add more information similar to namenode UI, > including, > * Static info (version, block pool and cluster ID) > * Block pools info (BP IDs, namenode address, actor states) > * Storage info (Volumes, capacity used, reserved, left) > * Utilities (logs) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10493) Add links to datanode web UI in namenode datanodes page
Weiwei Yang created HDFS-10493: -- Summary: Add links to datanode web UI in namenode datanodes page Key: HDFS-10493 URL: https://issues.apache.org/jira/browse/HDFS-10493 Project: Hadoop HDFS Issue Type: Improvement Components: namenode, ui Reporter: Weiwei Yang HDFS-10440 makes some improvements to datanode UI, it will be good to provide links from namenode datanodes information page to individual datanode UI, to check more datanode information easily. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10486) "Cannot start secure datanode with unprivileged HTTP ports" should give config param
[
https://issues.apache.org/jira/browse/HDFS-10486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317702#comment-15317702
]
Hadoop QA commented on HDFS-10486:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 5s {color}
| {color:red} Docker failed to build yetus/hadoop:2c91fd8. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808540/HDFS-10486.001.patch |
| JIRA Issue | HDFS-10486 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15674/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> "Cannot start secure datanode with unprivileged HTTP ports" should give
> config param
>
>
> Key: HDFS-10486
> URL: https://issues.apache.org/jira/browse/HDFS-10486
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Assignee: Yiqun Lin
>Priority: Trivial
> Attachments: HDFS-10486.001.patch
>
>
> The "Cannot start secure datanode with unprivileged HTTP ports" error should
> really give users a hint as to which parameter should get changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10486) "Cannot start secure datanode with unprivileged HTTP ports" should give config param
[ https://issues.apache.org/jira/browse/HDFS-10486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-10486: - Attachment: HDFS-10486.001.patch > "Cannot start secure datanode with unprivileged HTTP ports" should give > config param > > > Key: HDFS-10486 > URL: https://issues.apache.org/jira/browse/HDFS-10486 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, security >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Yiqun Lin >Priority: Trivial > Attachments: HDFS-10486.001.patch > > > The "Cannot start secure datanode with unprivileged HTTP ports" error should > really give users a hint as to which parameter should get changed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10486) "Cannot start secure datanode with unprivileged HTTP ports" should give config param
[ https://issues.apache.org/jira/browse/HDFS-10486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-10486: - Status: Patch Available (was: Open) Attach a simple patch for this, can the patch satisfy for you, [~aw]? > "Cannot start secure datanode with unprivileged HTTP ports" should give > config param > > > Key: HDFS-10486 > URL: https://issues.apache.org/jira/browse/HDFS-10486 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, security >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Yiqun Lin >Priority: Trivial > Attachments: HDFS-10486.001.patch > > > The "Cannot start secure datanode with unprivileged HTTP ports" error should > really give users a hint as to which parameter should get changed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10486) "Cannot start secure datanode with unprivileged HTTP ports" should give config param
[ https://issues.apache.org/jira/browse/HDFS-10486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin reassigned HDFS-10486: Assignee: Yiqun Lin > "Cannot start secure datanode with unprivileged HTTP ports" should give > config param > > > Key: HDFS-10486 > URL: https://issues.apache.org/jira/browse/HDFS-10486 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, security >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Yiqun Lin >Priority: Trivial > Attachments: HDFS-10486.001.patch > > > The "Cannot start secure datanode with unprivileged HTTP ports" error should > really give users a hint as to which parameter should get changed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317653#comment-15317653
]
Hadoop QA commented on HDFS-10458:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s {color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} branch-2.7 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} branch-2.7 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} branch-2.7 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s
{color} | {color:green} branch-2.7 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2630 line(s) that end in whitespace. Use
git apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 1m 4s
{color} | {color:red} The patch 70 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 46s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 7s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_101. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s
{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 145m 44s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| JDK v1.7.0_101 Failed junit tests |
hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.TestRollingUpgrade |
| |
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317645#comment-15317645
]
Hadoop QA commented on HDFS-10375:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
40s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 58m 24s
{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 23s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808511/HDFS-10375.004.patch |
| JIRA Issue | HDFS-10375 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ef434b3411f4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6de9213 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15672/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15672/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch,
> HDFS-10375.004.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
>
[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
[ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317644#comment-15317644 ] Colin Patrick McCabe commented on HDFS-10301: - Sorry for the slow reply. I was on vacation. Like I said earlier, I am -1 on patch v4 because adding new RPCs is bad for NN scalability. I also think it's a much larger patch than needed. It doesn't make sense as an interim solution. Why don't we commit v5 and discuss improvements in a follow-on JIRA? So far there is no concrete argument against it other than the fact that it doesn't remove zombie storages in the case where BRs are interleaved. But we already know that BR interleaving is an extremely rare corner case-- otherwise you can bet that this JIRA would have attracted a lot more attention. > BlockReport retransmissions may lead to storages falsely being declared > zombie if storage report processing happens out of order > > > Key: HDFS-10301 > URL: https://issues.apache.org/jira/browse/HDFS-10301 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.1 >Reporter: Konstantin Shvachko >Assignee: Colin Patrick McCabe >Priority: Critical > Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, > HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.01.patch, > HDFS-10301.sample.patch, zombieStorageLogs.rtf > > > When NameNode is busy a DataNode can timeout sending a block report. Then it > sends the block report again. Then NameNode while process these two reports > at the same time can interleave processing storages from different reports. > This screws up the blockReportId field, which makes NameNode think that some > storages are zombie. Replicas from zombie storages are immediately removed, > causing missing blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10467) Router-based HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317636#comment-15317636 ] He Tianyi commented on HDFS-10467: -- +1. This also reduces latency for first request from client (no failover on client-side, and router can memorize current active peer). > Router-based HDFS federation > > > Key: HDFS-10467 > URL: https://issues.apache.org/jira/browse/HDFS-10467 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Affects Versions: 2.7.2 >Reporter: Inigo Goiri > Attachments: HDFS Router Federation.pdf, > HDFS-Router-Federation-Prototype.patch > > > Add a Router to provide a federated view of multiple HDFS clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9924) [umbrella] Asynchronous HDFS Access
[ https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317629#comment-15317629 ] Colin Patrick McCabe commented on HDFS-9924: +1 for a feature branch > [umbrella] Asynchronous HDFS Access > --- > > Key: HDFS-9924 > URL: https://issues.apache.org/jira/browse/HDFS-9924 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Reporter: Tsz Wo Nicholas Sze >Assignee: Xiaobing Zhou > Attachments: AsyncHdfs20160510.pdf > > > This is an umbrella JIRA for supporting Asynchronous HDFS Access. > Currently, all the API methods are blocking calls -- the caller is blocked > until the method returns. It is very slow if a client makes a large number > of independent calls in a single thread since each call has to wait until the > previous call is finished. It is inefficient if a client needs to create a > large number of threads to invoke the calls. > We propose adding a new API to support asynchronous calls, i.e. the caller is > not blocked. The methods in the new API immediately return a Java Future > object. The return value can be obtained by the usual Future.get() method. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10467) Router-based HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317626#comment-15317626 ] Inigo Goiri commented on HDFS-10467: Another advantage about the proposed approach is that the Routers take care of the fail over of the Namenodes so it simplifies that in the Client side. Indirectly, this is using the approach that leverages centralized information which was discarded in HDFS-7858. This approach was discarded in that context but I think this is reasonable to do here. > Router-based HDFS federation > > > Key: HDFS-10467 > URL: https://issues.apache.org/jira/browse/HDFS-10467 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Affects Versions: 2.7.2 >Reporter: Inigo Goiri > Attachments: HDFS Router Federation.pdf, > HDFS-Router-Federation-Prototype.patch > > > Add a Router to provide a federated view of multiple HDFS clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10485) Fix findbugs warning in FSEditLog.java
[
https://issues.apache.org/jira/browse/HDFS-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-10485:
-
Attachment: HDFS-10485.01.patch
Attaching the same patch to run precommit job on trunk.
> Fix findbugs warning in FSEditLog.java
> --
>
> Key: HDFS-10485
> URL: https://issues.apache.org/jira/browse/HDFS-10485
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-10485.01.branch-2.patch, HDFS-10485.01.patch,
> HDFS-10485.02.branch-2.patch, HDFS-10485.03.branch-2.patch
>
>
> Found 1 findbugs warning when creating a patch for branch-2 in HDFS-10341
> (https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html)
> {noformat}
> Inconsistent synchronization of
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.numTransactionsBatchedInSync;
> locked 50% of time
> Bug type IS2_INCONSISTENT_SYNC (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.FSEditLog
> Field
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.numTransactionsBatchedInSync
> Synchronized 50% of the time
> Unsynchronized access at FSEditLog.java:[line 676]
> Unsynchronized access at FSEditLog.java:[line 676]
> Synchronized access at FSEditLog.java:[line 1254]
> Synchronized access at FSEditLog.java:[line 716]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10485) Fix findbugs warning in FSEditLog.java
[
https://issues.apache.org/jira/browse/HDFS-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317614#comment-15317614
]
Akira AJISAKA commented on HDFS-10485:
--
bq. If the patch applies to trunk we should just commit it to trunk and
branch-2.
Agreed. The patch applies to trunk cleanly.
> Fix findbugs warning in FSEditLog.java
> --
>
> Key: HDFS-10485
> URL: https://issues.apache.org/jira/browse/HDFS-10485
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-10485.01.branch-2.patch,
> HDFS-10485.02.branch-2.patch, HDFS-10485.03.branch-2.patch
>
>
> Found 1 findbugs warning when creating a patch for branch-2 in HDFS-10341
> (https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html)
> {noformat}
> Inconsistent synchronization of
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.numTransactionsBatchedInSync;
> locked 50% of time
> Bug type IS2_INCONSISTENT_SYNC (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.FSEditLog
> Field
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.numTransactionsBatchedInSync
> Synchronized 50% of the time
> Unsynchronized access at FSEditLog.java:[line 676]
> Unsynchronized access at FSEditLog.java:[line 676]
> Synchronized access at FSEditLog.java:[line 1254]
> Synchronized access at FSEditLog.java:[line 716]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10485) Fix findbugs warning in FSEditLog.java
[
https://issues.apache.org/jira/browse/HDFS-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-10485:
-
Summary: Fix findbugs warning in FSEditLog.java (was: Fix findbugs warning
in FSEditLog.java in branch-2)
> Fix findbugs warning in FSEditLog.java
> --
>
> Key: HDFS-10485
> URL: https://issues.apache.org/jira/browse/HDFS-10485
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-10485.01.branch-2.patch,
> HDFS-10485.02.branch-2.patch, HDFS-10485.03.branch-2.patch
>
>
> Found 1 findbugs warning when creating a patch for branch-2 in HDFS-10341
> (https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html)
> {noformat}
> Inconsistent synchronization of
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.numTransactionsBatchedInSync;
> locked 50% of time
> Bug type IS2_INCONSISTENT_SYNC (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.FSEditLog
> Field
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.numTransactionsBatchedInSync
> Synchronized 50% of the time
> Unsynchronized access at FSEditLog.java:[line 676]
> Unsynchronized access at FSEditLog.java:[line 676]
> Synchronized access at FSEditLog.java:[line 1254]
> Synchronized access at FSEditLog.java:[line 716]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10485) Fix findbugs warning in FSEditLog.java in branch-2
[
https://issues.apache.org/jira/browse/HDFS-10485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317607#comment-15317607
]
Arpit Agarwal commented on HDFS-10485:
--
+1 thanks [~ajisakaa].
If the patch applies to trunk we should just commit it to trunk and branch-2. I
am changing the jira title, feel free to change it back if this is not
applicable to trunk.
> Fix findbugs warning in FSEditLog.java in branch-2
> --
>
> Key: HDFS-10485
> URL: https://issues.apache.org/jira/browse/HDFS-10485
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-10485.01.branch-2.patch,
> HDFS-10485.02.branch-2.patch, HDFS-10485.03.branch-2.patch
>
>
> Found 1 findbugs warning when creating a patch for branch-2 in HDFS-10341
> (https://builds.apache.org/job/PreCommit-HDFS-Build/15639/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html)
> {noformat}
> Inconsistent synchronization of
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.numTransactionsBatchedInSync;
> locked 50% of time
> Bug type IS2_INCONSISTENT_SYNC (click for details)
> In class org.apache.hadoop.hdfs.server.namenode.FSEditLog
> Field
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.numTransactionsBatchedInSync
> Synchronized 50% of the time
> Unsynchronized access at FSEditLog.java:[line 676]
> Unsynchronized access at FSEditLog.java:[line 676]
> Synchronized access at FSEditLog.java:[line 1254]
> Synchronized access at FSEditLog.java:[line 716]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317594#comment-15317594
]
Andrew Wang commented on HDFS-10375:
004 patch LGTM, will commit once Jenkins comes back clean. Thanks for the patch
Jiayi and John for doing reviews.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch,
> HDFS-10375.004.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10490) Client may never recovery replica after a timeout during sending packet
[
https://issues.apache.org/jira/browse/HDFS-10490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317587#comment-15317587
]
He Tianyi commented on HDFS-10490:
--
Thanks for pointing that out, [~kihwal].
I think the case is a little different here. The catch is whether DataNode
flushed metafile header to disk or not.
If not (as in this case, output stream for metafile is not properly closed yet,
for there is no way for DataNode to actively detect a network problem),
{{BlockSender}} will see a 0-length metafile (not even containing header), then
use (NULL, 512) for checksuming, client will still fail, given that
configurated bytesPerChecksum != 512.
> Client may never recovery replica after a timeout during sending packet
> ---
>
> Key: HDFS-10490
> URL: https://issues.apache.org/jira/browse/HDFS-10490
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.6.0
>Reporter: He Tianyi
> Attachments: HDFS-10490.patch
>
>
> For newly created replica, a meta file is created in constructor of
> {{BlockReceiver}} (for {{WRITE_BLOCK}} op). Its header will be written lazily
> (buffered in memory first by {{BufferedOutputStream}}).
> If following packets fail to deliver (e.g. in extreme network condition),
> the header may never get flush until closed.
> However, {{BlockReceiver}} will not call close until block receiving is
> finished or exception(s) encountered. Also in extreme network condition, both
> RST & FIN may not deliver in time.
> In this case, if client tries to initiates a {{transferBlock}} to a new
> datanode (in {{addDatanode2ExistingPipeline}}), existing datanode will see an
> empty meta if its {{BlockReceiver}} did not close in time.
> Then, after HDFS-3429, a default {{DataChecksum}} (NULL, 512) will be used
> during transfer. So when client then tries to recover pipeline after
> completely transferred, it may encounter the following exception:
> {noformat}
> java.io.IOException: Client requested checksum DataChecksum(type=CRC32C,
> chunkSize=4096) when appending to an existing block with different chunk
> size: DataChecksum(type=NULL, chunkSize=512)
> at
> org.apache.hadoop.hdfs.server.datanode.ReplicaInPipeline.createStreams(ReplicaInPipeline.java:230)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:226)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:798)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:166)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:76)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:243)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This will repeat, until exhausted by datanode replacement policy.
> Also to note that, with bad luck (like I), 20k clients are all doing this.
> It's to some extend a DDoS attack to NameNode (because of
> getAdditionalDataNode calls).
> I suggest we flush immediately after header is written, preventing anybody
> from seeing empty meta file for avoiding the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-8872) Reporting of missing blocks is different in fsck and namenode ui/metasave
[
https://issues.apache.org/jira/browse/HDFS-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317548#comment-15317548
]
Ming Ma edited comment on HDFS-8872 at 6/7/16 12:26 AM:
Thanks [~shahrs87]. Your suggestion to distinguish decommissioned from
decommissioning makes sense. Given liveness state is separated from admin
state, it appears the suggestion is "a block should be considered missing if
and only if {{live, normal}} + {{live, decommissioning}} == 0". The live
attribute is implicit given fsck only gets the list of live replicas; but it
might be useful to call it out.
If that is the case, both fsck and jmx metrics need to be changed. Also
depending on how you interpret it, it seems more like incompatible change than
just bug fix. That will decide if we want to fix it only in trunk.
was (Author: mingma):
Thanks [~shahrs87]. Your suggestion to distinguish decommissioned from
decommissioning makes sense. Given liveness state is separated from admin
state, it appears the suggestion is "a block should be considered missing if
and only if {live, normal} + {live, decommissioning} == 0". The live attribute
is implicit given fsck only gets the list of live replicas; but it might be
useful to call it out.
If that is the case, both fsck and jmx metrics need to be changed. Also
depending on how you interpret it, it seems more like incompatible change than
just bug fix. That will decide if we want to fix it only in trunk.
> Reporting of missing blocks is different in fsck and namenode ui/metasave
> -
>
> Key: HDFS-8872
> URL: https://issues.apache.org/jira/browse/HDFS-8872
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>
> Namenode ui and metasave will not report a block as missing if the only
> replica is on decommissioning/decomissioned node while fsck will show it as
> MISSING.
> Since decommissioned node can be formatted/removed anytime, we can actually
> lose the block.
> Its better to alert on namenode ui if the only copy is on
> decomissioned/decommissioning node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8872) Reporting of missing blocks is different in fsck and namenode ui/metasave
[
https://issues.apache.org/jira/browse/HDFS-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317548#comment-15317548
]
Ming Ma commented on HDFS-8872:
---
Thanks [~shahrs87]. Your suggestion to distinguish decommissioned from
decommissioning makes sense. Given liveness state is separated from admin
state, it appears the suggestion is "a block should be considered missing if
and only if {live, normal} + {live, decommissioning} == 0". The live attribute
is implicit given fsck only gets the list of live replicas; but it might be
useful to call it out.
If that is the case, both fsck and jmx metrics need to be changed. Also
depending on how you interpret it, it seems more like incompatible change than
just bug fix. That will decide if we want to fix it only in trunk.
> Reporting of missing blocks is different in fsck and namenode ui/metasave
> -
>
> Key: HDFS-8872
> URL: https://issues.apache.org/jira/browse/HDFS-8872
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>
> Namenode ui and metasave will not report a block as missing if the only
> replica is on decommissioning/decomissioned node while fsck will show it as
> MISSING.
> Since decommissioned node can be formatted/removed anytime, we can actually
> lose the block.
> Its better to alert on namenode ui if the only copy is on
> decomissioned/decommissioning node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: HDFS-10375.004.patch
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch,
> HDFS-10375.004.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10440) Improve DataNode web UI
[
https://issues.apache.org/jira/browse/HDFS-10440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317531#comment-15317531
]
Weiwei Yang commented on HDFS-10440:
I don't think the 2 failed tests are related, I can run them successfully
locally,
{quote}
---
Test set: org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
---
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.011 sec -
in org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
---
Test set: org.apache.hadoop.hdfs.TestRollingUpgrade
---
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 152.713 sec -
in org.apache.hadoop.hdfs.TestRollingUpgrade
{quote}
Found an opening JIRA that similar to 2nd issue : HDFS-9664.
> Improve DataNode web UI
> ---
>
> Key: HDFS-10440
> URL: https://issues.apache.org/jira/browse/HDFS-10440
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.5.0, 2.6.0, 2.7.0
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-10440.001.patch, HDFS-10440.002.patch,
> datanode_2nns.html.002.jpg, datanode_html.001.jpg,
> datanode_loading_err.002.jpg, datanode_utilities.001.jpg,
> datanode_utilities.002.jpg, dn_web_ui_mockup.jpg
>
>
> At present, datanode web UI doesn't have much information except for node
> name and port. Propose to add more information similar to namenode UI,
> including,
> * Static info (version, block pool and cluster ID)
> * Block pools info (BP IDs, namenode address, actor states)
> * Storage info (Volumes, capacity used, reserved, left)
> * Utilities (logs)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317517#comment-15317517
]
Hadoop QA commented on HDFS-10375:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 26s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1
new + 6 unchanged - 0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 58m 35s
{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 27s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808496/HDFS-10375.003.patch |
| JIRA Issue | HDFS-10375 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f3161f01f9c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a3f78d8 |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15670/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15670/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15670/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority:
[jira] [Commented] (HDFS-9353) Code and comment mismatch in JavaKeyStoreProvider
[
https://issues.apache.org/jira/browse/HDFS-9353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317502#comment-15317502
]
Daniel Templeton commented on HDFS-9353:
I'm OK with just deleting the comment.
> Code and comment mismatch in JavaKeyStoreProvider
> ---
>
> Key: HDFS-9353
> URL: https://issues.apache.org/jira/browse/HDFS-9353
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: nijel
>Assignee: Andras Bokor
>Priority: Trivial
>
> In
> org.apache.hadoop.crypto.key.JavaKeyStoreProvider.JavaKeyStoreProvider(URI
> uri, Configuration conf) throws IOException
> The comment mentioned is
> {code}
> // Get the password file from the conf, if not present from the user's
> // environment var
> {code}
> But the code takes the value form ENV first
> I think this make sense since the user can pass the ENV for a particular run.
> My suggestion is to change the comment
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317484#comment-15317484
]
Hudson commented on HDFS-10458:
---
SUCCESS: Integrated in Hadoop-trunk-Commit #9916 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9916/])
HDFS-10458. getFileEncryptionInfo should return quickly for (zhz: rev
6de9213df111a9a4ed875db995d67af72d08a798)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirEncryptionZoneOp.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionZoneManager.java
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch,
> HDFS-10458.03.patch, HDFS-10458.04.patch, HDFS-10458.05.patch,
> HDFSA-10458.01.patch, HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10468) HDFS read ends up ignoring an interrupt
[
https://issues.apache.org/jira/browse/HDFS-10468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317480#comment-15317480
]
Hadoop QA commented on HDFS-10468:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s
{color} | {color:red} hadoop-hdfs-project: The patch generated 1 new + 125
unchanged - 2 fixed = 126 total (was 127) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 59s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 109m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestAsyncHDFSWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808478/HDFS-10468.003.patch |
| JIRA Issue | HDFS-10468 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ab32f06023b3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4a1cedc |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15669/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15669/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
[jira] [Updated] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10458:
-
Target Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1 (was: 2.6.5)
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch,
> HDFS-10458.03.patch, HDFS-10458.04.patch, HDFS-10458.05.patch,
> HDFSA-10458.01.patch, HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10458:
-
Attachment: HDFS-10458-branch-2.7.00.patch
Back porting to 2.7 not clean. Attaching patch to trigger Jenkins.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10458-branch-2.7.00.patch, HDFS-10458.00.patch,
> HDFS-10458.03.patch, HDFS-10458.04.patch, HDFS-10458.05.patch,
> HDFSA-10458.01.patch, HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317440#comment-15317440
]
John Zhuge commented on HDFS-10375:
---
Looks good. BTW, it shoulda been named 004.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317410#comment-15317410
]
Jiayi Zhou edited comment on HDFS-10375 at 6/6/16 10:31 PM:
I've merged them into 1 commit now. How about this new patch 003?
was (Author: clouderajiayi):
HI've merged them into 1 commit now. How about this new patch 003?
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317410#comment-15317410
]
Jiayi Zhou commented on HDFS-10375:
---
HI've merged them into 1 commit now. How about this new patch 003?
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317412#comment-15317412
]
Hadoop QA commented on HDFS-10458:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1
new + 4 unchanged - 0 fixed = 5 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 52s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 22s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.TestAsyncHDFSWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808443/HDFS-10458.05.patch |
| JIRA Issue | HDFS-10458 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 29ca33fd3663 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 35f255b |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15667/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15667/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15667/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/15667/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15667/console |
|
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: (was: HDFS-10375.003.patch)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: HDFS-10375.003.patch
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-10488:
-
Status: Open (was: Patch Available)
> WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating
> files/directories without specifying permissions
> --
>
> Key: HDFS-10488
> URL: https://issues.apache.org/jira/browse/HDFS-10488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch,
> HDFS-10488.patch
>
>
> WebHDFS methods for creating file/directories are always creating it with 755
> permissions as default, even ignoring any configured
> *fs.permissions.umask-mode* in the case of directories.
> Dfs CLI, however, applies the configured umask to 777 permission for
> directories, or 666 permission for files.
> Example below shows the different behaviour when creating directory via CLI
> and WebHDFS:
> {noformat}
> 1) Creating a directory under '/test/' as 'test-user'. Configured
> fs.permissions.umask-mode is 000:
> $ sudo -u test-user hdfs dfs -mkdir /test/test-user1
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user1
> # file: /test/test-user1
> # owner: test-user
> # group: supergroup
> user::rwx
> group::rwx
> other::rwx
> 4) Doing the same via WebHDFS does not get the proper ACLs:
> $ curl -i -X PUT
> "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user=MKDIRS;
>
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user2
> # file: /test/test-user2
> # owner: test-user
> # group: supergroup
> user::rwx
> group::r-x
> other::r-x
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317387#comment-15317387
]
Chris Nauroth commented on HDFS-10488:
--
-1 for the proposed change.
The umask ({{fs.permissions.umask-mode}}) is a concept that is applied at the
client side by individual applications, usually via their usage of the
FileSystem subclasses that implement a particular file system client. The
umask is not applied by the API/protocol layer such as WebHDFS or NameNode RPC.
As such, the behavior of the shell, which applies umask, is not always going
to look consistent with the behavior of a direct curl WebHDFS call, which does
not apply the umask.
Using the shell to access WebHDFS gives consistent results, because the logic
of the WebHdfsFileSystem class used by the shell will apply the umask.
If this patch were committed, then it would become basically impossible to
create files and directories with absolute permissions through WebHDFS. For
example, suppose {{fs.permissions.umask-mode}} is set to 022, but an individual
application has a desire to create a file with 775 permissions. This wouldn't
work as expected, because server-side enforcement of the umask would restrict
permissions on the resulting file to 755. The only way to work around this
would be to reconfigure {{fs.permissions.umask-mode}} and restart the NameNode,
which isn't operationally desirable. Worse than that, this would likely have
the long-term effect of reducing {{fs.permissions.umask-mode}} to lowest common
denominator, perhaps even 000, to accommodate all possible permissions at file
creation time, thus weakening the benefit of umask as applied by client
applications like the shell.
As a final point against this change, please note that it could be considered
backwards-incompatible. In my example above trying to create a file with 775
permissions, but the server-side umask forcing it to 755, it means that
subsequent write actions by users in the same group will be unauthorized. This
may break certain workflows.
The area where there is a possibility for change is documentation to help raise
user awareness of this. That could potentially go into the HDFS Permissions
Guide page or the WebHDFS REST API page, or perhaps some combination of both.
I would be happy to help review and +1 documentation changes.
[~wchevreuil], despite my -1, thank you for writing up your experience with
this and posting a patch. If you'd like to proceed with a documentation patch,
please let me know, and I'll assign the issue to you.
> WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating
> files/directories without specifying permissions
> --
>
> Key: HDFS-10488
> URL: https://issues.apache.org/jira/browse/HDFS-10488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch,
> HDFS-10488.patch
>
>
> WebHDFS methods for creating file/directories are always creating it with 755
> permissions as default, even ignoring any configured
> *fs.permissions.umask-mode* in the case of directories.
> Dfs CLI, however, applies the configured umask to 777 permission for
> directories, or 666 permission for files.
> Example below shows the different behaviour when creating directory via CLI
> and WebHDFS:
> {noformat}
> 1) Creating a directory under '/test/' as 'test-user'. Configured
> fs.permissions.umask-mode is 000:
> $ sudo -u test-user hdfs dfs -mkdir /test/test-user1
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user1
> # file: /test/test-user1
> # owner: test-user
> # group: supergroup
> user::rwx
> group::rwx
> other::rwx
> 4) Doing the same via WebHDFS does not get the proper ACLs:
> $ curl -i -X PUT
> "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user=MKDIRS;
>
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user2
> # file: /test/test-user2
> # owner: test-user
> # group: supergroup
> user::rwx
> group::r-x
> other::r-x
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317358#comment-15317358
]
John Zhuge edited comment on HDFS-10375 at 6/6/16 10:09 PM:
[~clouderajiayi] The patch 003 is wrong because it is incremental to 002. It
should be a diff from the trunk. It seems your local branch is ahead of trunk
by 2 commits : the older one for patch 002 and the newer one for patch 003.
Please merge them into 1 commit by running {{git rebase -i HEAD~2}} (fixup the
newer commit) then run format-patch again.
was (Author: jzhuge):
[~clouderajiayi] The patch 003 is wrong because it is incremental to 002. It
should be a diff from the trunk. It seems your local branch is ahead of trunk
by 2 commits : the older one for patch 002 and the newer one for patch 003. In
this case, please run {{git format-patch --stdout HEAD~2 >
HDFS-10375.NNN.patch}}.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317358#comment-15317358
]
John Zhuge commented on HDFS-10375:
---
[~clouderajiayi] The patch 003 is wrong because it is incremental to 002. It
should be a diff from the trunk. It seems your local branch is ahead of trunk
by 2 commits : the older one for patch 002 and the newer one for patch 003. In
this case, please run {{git format-patch --stdout HEAD~2 >
HDFS-10375.NNN.patch}}.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10491) libhdfs++: Implement GetFsStats
[
https://issues.apache.org/jira/browse/HDFS-10491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317350#comment-15317350
]
Hadoop QA commented on HDFS-10491:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 7m 43s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
9s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 47s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 7s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 3m 48s {color} |
{color:red} hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_91 with JDK
v1.8.0_91 generated 1 new + 67 unchanged - 1 fixed = 68 total (was 68) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 42s
{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 24s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 31s
{color} | {color:green} hadoop-hdfs-native-client in the patch passed with JDK
v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 58s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed CTEST tests |
test_libhdfs_mini_stress_hdfspp_test_shim_static |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808473/HDFS-10491.HDFS-8707.000.patch
|
| JIRA Issue | HDFS-10491 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 887d8dbb9699 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / e8f326a |
| Default Java | 1.7.0_101 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_91
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 |
| cc |
https://builds.apache.org/job/PreCommit-HDFS-Build/15668/artifact/patchprocess/diff-compile-cc-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_91.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/15668/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_91.txt
|
| CTEST logs |
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317341#comment-15317341
]
Jiayi Zhou commented on HDFS-10375:
---
Hi John, how can I run this Hadoop QA again? I think I have fixed the problem
and it should work properly now.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10468) HDFS read ends up ignoring an interrupt
[ https://issues.apache.org/jira/browse/HDFS-10468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-10468: - Attachment: HDFS-10468.003.patch Thanks a lot for the review, [~iwasakims]. Yes, to change the log level to debug should be better here: the upper level application will get the exception anyway. Upload a new patch to address your comments. > HDFS read ends up ignoring an interrupt > --- > > Key: HDFS-10468 > URL: https://issues.apache.org/jira/browse/HDFS-10468 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Jing Zhao > Attachments: HDFS-10468.000.patch, HDFS-10468.001.patch, > HDFS-10468.002.patch, HDFS-10468.003.patch, log > > > If an interrupt comes in during an HDFS read - it looks like HDFS ends up > ignoring it (handling it), and retries the read after an interval. > An interrupt should result in the read being cancelled, with an > InterruptedException being thrown. > Similarly - if an HDFS op is started with the interrupt status on the thread > set, an InterruptedException should be thrown. > cc [~jingzhao] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10491) libhdfs++: Implement GetFsStats
[ https://issues.apache.org/jira/browse/HDFS-10491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-10491: - Attachment: HDFS-10491.HDFS-8707.000.patch New patch > libhdfs++: Implement GetFsStats > --- > > Key: HDFS-10491 > URL: https://issues.apache.org/jira/browse/HDFS-10491 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein > Attachments: HDFS-10491.HDFS-8707.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10491) libhdfs++: Implement GetFsStats
[ https://issues.apache.org/jira/browse/HDFS-10491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-10491: - Status: Patch Available (was: Open) > libhdfs++: Implement GetFsStats > --- > > Key: HDFS-10491 > URL: https://issues.apache.org/jira/browse/HDFS-10491 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein > Attachments: HDFS-10491.HDFS-8707.000.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10492) libhdfs++: Clean up minidfscluster tests
James Clampffer created HDFS-10492: -- Summary: libhdfs++: Clean up minidfscluster tests Key: HDFS-10492 URL: https://issues.apache.org/jira/browse/HDFS-10492 Project: Hadoop HDFS Issue Type: Sub-task Reporter: James Clampffer Assignee: James Clampffer A couple things need to be fixed with the minidfscluster tests -Tests that are using hdfs_ext shouldn't be living in libhdfs-tests, any test in there should be able to be shared between both libraries. -Tests added in HDFS-9890 relies on NDEBUG to turn on error simulation, ideally these should have some other switch so we can run error simulation on release builds. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317234#comment-15317234
]
Zhe Zhang commented on HDFS-10458:
--
Thanks [~shv]. Will commit to trunk ~ branch-2.6 after the latest Jenkins run.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317205#comment-15317205
]
Konstantin Shvachko commented on HDFS-10458:
+1. Tests passed locally.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10486) "Cannot start secure datanode with unprivileged HTTP ports" should give config param
[ https://issues.apache.org/jira/browse/HDFS-10486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317052#comment-15317052 ] Steve Loughran commented on HDFS-10486: --- why not link to a wiki entry on the topic? > "Cannot start secure datanode with unprivileged HTTP ports" should give > config param > > > Key: HDFS-10486 > URL: https://issues.apache.org/jira/browse/HDFS-10486 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, security >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Priority: Trivial > > The "Cannot start secure datanode with unprivileged HTTP ports" error should > really give users a hint as to which parameter should get changed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10484) Can not read file from java.io.IOException: Need XXX bytes, but only YYY bytes available
[ https://issues.apache.org/jira/browse/HDFS-10484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HDFS-10484. --- Resolution: Cannot Reproduce > Can not read file from java.io.IOException: Need XXX bytes, but only YYY > bytes available > - > > Key: HDFS-10484 > URL: https://issues.apache.org/jira/browse/HDFS-10484 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.0.0-alpha > Environment: Cloudera 4.1.2, hadoop-hdfs-2.0.0+552-1.cdh4.1.2.p0.27 >Reporter: pt > > We are running CDH 4.1.2 distro and trying to read file from HDFS. It ends up > with exception @datanode saying > 2016-06-02 10:43:26,354 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > DatanodeRegistration(X.X.X.X, > storageID=DS-404876644-X.X.X.X-50010-1462535537579, infoPort=50075, > ipcPort=50020, storageInfo=lv=-40;cid=cluster18;nsid=2115086255;c=0):Got > exception while serving > BP-2091182050-X.X.X.X-1358362115729:blk_5037101550399368941_420502314 to > /X.X.X.X:58614 > java.io.IOException: Need 10172416 bytes, but only 10072576 bytes available > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.waitForMinLength(BlockSender.java:387) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:189) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:268) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:88) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:63) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219) > at java.lang.Thread.run(Thread.java:662) > 2016-06-02 10:43:26,354 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: > app112.rutarget.ru:50010:DataXceiver error processing READ_BLOCK operation > src: /X.X.X.X:58614 dest: /X.X.X.X:50010 > java.io.IOException: Need 10172416 bytes, but only 10072576 bytes available > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.waitForMinLength(BlockSender.java:387) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:189) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:268) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:88) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:63) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219) > at java.lang.Thread.run(Thread.java:662) > FSCK shows file as being open for write, however hdfs client that handles > writes to this file closed it long time ago -- so file stucked in RBW for a > few last days. How can we get actual data block in this case? I found only > binary .meta file on datanode but not actual block with data. > -- -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10484) Can not read file from java.io.IOException: Need XXX bytes, but only YYY bytes available
[ https://issues.apache.org/jira/browse/HDFS-10484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317049#comment-15317049 ] Steve Loughran commented on HDFS-10484: --- That is a really old version of Hadoop. It's not going to be anything that the Hadoop open source team are going to look at directly. Sorry you're either going to have to take it up with cloudera, or upgrade to a recent Apache release and see if the problem goes away. Closing as invalid, again, with apologies. see: https://wiki.apache.org/hadoop/InvalidJiraIssues > Can not read file from java.io.IOException: Need XXX bytes, but only YYY > bytes available > - > > Key: HDFS-10484 > URL: https://issues.apache.org/jira/browse/HDFS-10484 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.0.0-alpha > Environment: Cloudera 4.1.2, hadoop-hdfs-2.0.0+552-1.cdh4.1.2.p0.27 >Reporter: pt > > We are running CDH 4.1.2 distro and trying to read file from HDFS. It ends up > with exception @datanode saying > 2016-06-02 10:43:26,354 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: > DatanodeRegistration(X.X.X.X, > storageID=DS-404876644-X.X.X.X-50010-1462535537579, infoPort=50075, > ipcPort=50020, storageInfo=lv=-40;cid=cluster18;nsid=2115086255;c=0):Got > exception while serving > BP-2091182050-X.X.X.X-1358362115729:blk_5037101550399368941_420502314 to > /X.X.X.X:58614 > java.io.IOException: Need 10172416 bytes, but only 10072576 bytes available > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.waitForMinLength(BlockSender.java:387) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:189) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:268) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:88) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:63) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219) > at java.lang.Thread.run(Thread.java:662) > 2016-06-02 10:43:26,354 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: > app112.rutarget.ru:50010:DataXceiver error processing READ_BLOCK operation > src: /X.X.X.X:58614 dest: /X.X.X.X:50010 > java.io.IOException: Need 10172416 bytes, but only 10072576 bytes available > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.waitForMinLength(BlockSender.java:387) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:189) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:268) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:88) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:63) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219) > at java.lang.Thread.run(Thread.java:662) > FSCK shows file as being open for write, however hdfs client that handles > writes to this file closed it long time ago -- so file stucked in RBW for a > few last days. How can we get actual data block in this case? I found only > binary .meta file on datanode but not actual block with data. > -- -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317035#comment-15317035 ] Anu Engineer commented on HDFS-7240: Hi [~eddyxu] , Thank you for reviewing the design doc and comments. Please see my comments below. bq. Since Ozone is decided to use range partition, how would key / data distribution achieve balancing from initial state? For example, a user Foo runs Hive and creates 10GB of data, these data are distributed to up to 6 (containers) DNs? You bring up a very valid point. This was the most contentious issue in ozone world for a while. We originally went with hash partition schemes and secondary index because of these concerns. The issue (and very rightly so) with that approach was that secondary index is eventually consistent and makes it hard to use. So we switched over to this scheme. So our current thought is this, each of the containers will report -- size, number of operations and number of keys to SCM. This will allow SCM to balance the allocation of the key space. So if you have a large number of reads and writes, which are completely independent, then they will fill up the cluster/container space evenly. But we have an opposing requirement here, generally there is a locality of access in the namespace. So for most cases if you are reading and writing to a bucket, then it is most efficient to keep that data together. Now let us look at this specific case, if you have containers configured to say 2GB, then 10GB of data will map to 5 containers. So the model works out to 5 containers. These containers will be spread across a set of machines due to the SCM’s location choosing algorithms. bq. Would you explain what is the benefit of recovering failure pipeline by using a parallel writes to all 3 containers? It is not very clear in the design. The point I was trying to make is that pipeline relies on Quorum as defined by RSM. So if we decide to use this pipeline with RAFT, then I was just trying to make a point that pipeline can be broken, and we will not attempt to heal it. Please let me know if this makes sense. bq. How does ozone differentiate a recover write from a malicious (or buggy) re-write? Thanks for flagging this, right now we do not. We can always prevent it in the container layer. It is small extension to make, we can write to a temporary file and replace the original if and only if the hashes match. I will file a work item to fix this. bq. You mentioned that KMS/SCM separation is for future scalability. Do KMS / SCM maintains 1:1, 1:n or n:m relationship? Though it is not in this phase. I'd like to know whether it is considered. Btw, they are also Raft replicated? KSM:SCM has a n:m relationship. Even though in easiest deployment configuration it is 1:1. So yes it is defined that way. They are always Raft replicated. bq. The raft ring / leader is per-container? Yes, and No. Let me explain this a little more. If you think only in terms of RAFT protocol, then we have a RAFT leader is per machine set. That is, we are going to have a leader for 3 machines (assuming a 3 machine RAFT ring). Now let us switch over to a developer’s point of view. Someone like me who is writing code against containers thinks strictly in terms of containers. So from an ozone developers point of view, we have a Raft leader for a container. In other words, containers provide an abstraction that makes you think that RAFT protocol is for the container, whereas in reality it is a shared ring that is used by many containers that share those 3 machines. This might be something that we want to explore in greater depth during the call. bq. For pipeline, say if we have a pipeline A->B->C, if the data writes successfully on A->B, and the metadata Raft writes are succeed on B,C, IIUC, that is a What would be the result for a read request sent to A or C? I am going to walk thru this with little more details, so that we are all on the same page. What you are describing is a situation where the RAFT leader is either B or C (Since RAFT is an active leader protocol) and for the sake of this illustration let us assume that we are talking about 2 scenarios. One where data is written to leader and another datanode and scenario two, where data is written to followers but not to the leader. Let us look at both in greater detail. Case 1: Data is written to machines B (leader) and Machine A. But when RAFT commit happens, machine A is off-line and RAFT data is written to Machine B and Machine C. So we have situation where B is the only machine with metadata as well as data. We deal with this issue in two ways, one when the commit callback happens in C, C will check if it has the data block and since it does not, it will attempt to copy that block from either B or A. Also when A's RAFT ring comes back up it will catch up with the RAFT log and
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15317015#comment-15317015
]
Hadoop QA commented on HDFS-10375:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 28s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 28s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs: The patch generated 3
new + 0 unchanged - 6 fixed = 3 total (was 6) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 30s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 11s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s {color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 55s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808441/HDFS-10375.002.patch |
| JIRA Issue | HDFS-10375 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8f1853535fc5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 35f255b |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/15664/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/15664/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/15664/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/15664/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/15664/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/15664/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: HDFS-10375.003.patch
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch, HDFS-10375.003.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316995#comment-15316995
]
Wei-Chiu Chuang commented on HDFS-10488:
I believe he's not on the contributor list which is why he can't assign this to
himself.
> WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating
> files/directories without specifying permissions
> --
>
> Key: HDFS-10488
> URL: https://issues.apache.org/jira/browse/HDFS-10488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch,
> HDFS-10488.patch
>
>
> WebHDFS methods for creating file/directories are always creating it with 755
> permissions as default, even ignoring any configured
> *fs.permissions.umask-mode* in the case of directories.
> Dfs CLI, however, applies the configured umask to 777 permission for
> directories, or 666 permission for files.
> Example below shows the different behaviour when creating directory via CLI
> and WebHDFS:
> {noformat}
> 1) Creating a directory under '/test/' as 'test-user'. Configured
> fs.permissions.umask-mode is 000:
> $ sudo -u test-user hdfs dfs -mkdir /test/test-user1
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user1
> # file: /test/test-user1
> # owner: test-user
> # group: supergroup
> user::rwx
> group::rwx
> other::rwx
> 4) Doing the same via WebHDFS does not get the proper ACLs:
> $ curl -i -X PUT
> "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user=MKDIRS;
>
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user2
> # file: /test/test-user2
> # owner: test-user
> # group: supergroup
> user::rwx
> group::r-x
> other::r-x
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316976#comment-15316976
]
Wei-Chiu Chuang commented on HDFS-10488:
Hi [~wchevreuil] thanks for the great finding and the patch.
The fix itself looks good to me in general and I just have 1 question about the
fix and some quick comments regarding the test:
Regarding the fix:
I am a little confused by the following line: seems you always call
getDirFsPermission() regardless whether it's a dir or file?
{code}
np.setPermission(fullpath, permission.getDirFsPermission());
{code}
# The convention of HDFS unit testing is to start a MiniDFSCluster using
{{@Before}} annotation, and tear down the cluster using {{@After}}. You can
take a look at other HDFS test files for reference.
# Would you mind to add additional tests to verify it works as expected if the
mask {{fs.permissions.umask-mode}} is set differently? This will be helpful in
detecting other webhfs permission bugs if they exist.
# You have fixed the WebHDFS CREATESYMLINK operation. Could you also add tests
to verify this operation as well?
> WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating
> files/directories without specifying permissions
> --
>
> Key: HDFS-10488
> URL: https://issues.apache.org/jira/browse/HDFS-10488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch,
> HDFS-10488.patch
>
>
> WebHDFS methods for creating file/directories are always creating it with 755
> permissions as default, even ignoring any configured
> *fs.permissions.umask-mode* in the case of directories.
> Dfs CLI, however, applies the configured umask to 777 permission for
> directories, or 666 permission for files.
> Example below shows the different behaviour when creating directory via CLI
> and WebHDFS:
> {noformat}
> 1) Creating a directory under '/test/' as 'test-user'. Configured
> fs.permissions.umask-mode is 000:
> $ sudo -u test-user hdfs dfs -mkdir /test/test-user1
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user1
> # file: /test/test-user1
> # owner: test-user
> # group: supergroup
> user::rwx
> group::rwx
> other::rwx
> 4) Doing the same via WebHDFS does not get the proper ACLs:
> $ curl -i -X PUT
> "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user=MKDIRS;
>
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user2
> # file: /test/test-user2
> # owner: test-user
> # group: supergroup
> user::rwx
> group::r-x
> other::r-x
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316975#comment-15316975
]
John Zhuge commented on HDFS-10375:
---
You can upload multiple patches as long as they are all numbered sequentially
which you did.
On you local machines, just run {{git commit --amend}} then {{git format-patch
--stdout HEAD~ > HDFS-10375.NNN.patch}} (increment NNN each time).
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-10375:
---
Fix Version/s: (was: 2.8.0)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316964#comment-15316964
]
Jiayi Zhou commented on HDFS-10375:
---
So I only need to git commit again on my local machine?
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316960#comment-15316960
]
John Zhuge commented on HDFS-10375:
---
Thanks [~clouderajiayi] for contributing the patches. There is no need to
remove the the previous patch before adding a new one.
+1(non-binding) LGTM.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10375:
--
Hadoop Flags: Reviewed
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10458:
-
Attachment: HDFS-10458.05.patch
Updating patch to address unit test failures. Thanks [~shv] for the review.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFS-10458.05.patch, HDFSA-10458.01.patch,
> HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: HDFS-10375.002.patch
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.002.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: (was: HDFS-10375.001.patch)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Status: Patch Available (was: Open)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7941) hsync() not working
[ https://issues.apache.org/jira/browse/HDFS-7941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316948#comment-15316948 ] Johannes Herr commented on HDFS-7941: - The file may not appear to grow because the new length is not sent to the Namenode on hsync or because Block-compression is used and the data is still in the compress buffer. See http://stackoverflow.com/a/37664114/136247 for more details. > hsync() not working > --- > > Key: HDFS-7941 > URL: https://issues.apache.org/jira/browse/HDFS-7941 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.6.0 > Environment: HDP 2.2 running on Redhat >Reporter: Sverre Bakke > > When using SequenceFile.Writer and appending+syncing to file repeatedly, the > sync does not appear to work other than: > - once after writing headers > - when closing. > Imagine the following test case: > http://pastebin.com/Y9xysCRX > This code would append a new record every second and then immediately sync > it. One would also imagine that the file would grow for every append, > however, this does not happen. > After watching the behavior I have noticed that it only syncs the headers at > the very beginning (providing a file of 164 bytes) and then never again until > its closed. This despite it is asked to hsync() after every append. > Looking into the debug logs, this also claims the same behavior (executed the > provided code example and grepped for "sync"): > SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". > SLF4J: Defaulting to no-operation (NOP) logger implementation > SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further > details. > 2015-03-17 15:55:14 DEBUG ProtobufRpcEngine:253 - Call: fsync took 11ms > This was the only time the code ran fsync throughout the entire execution. > This has been tested (with similar result) for the following deployments: > - sequencefile with no compression > - sequencefile with record compression > - sequencefile with block compression > - textfile with no compression -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Status: Open (was: Patch Available)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10458) getFileEncryptionInfo should return quickly for non-encrypted cluster
[
https://issues.apache.org/jira/browse/HDFS-10458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316930#comment-15316930
]
Konstantin Shvachko commented on HDFS-10458:
Looks like in the end you will need to pass an empty list into
{{BatchedListEntries}} constructor.
> getFileEncryptionInfo should return quickly for non-encrypted cluster
> -
>
> Key: HDFS-10458
> URL: https://issues.apache.org/jira/browse/HDFS-10458
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, namenode
>Affects Versions: 2.6.0
>Reporter: Zhe Zhang
>Assignee: Zhe Zhang
> Attachments: HDFS-10458.00.patch, HDFS-10458.03.patch,
> HDFS-10458.04.patch, HDFSA-10458.01.patch, HDFSA-10458.02.patch
>
>
> {{FSDirectory#getFileEncryptionInfo}} always acquires {{readLock}} and checks
> if the path belongs to an EZ. For a busy system with potentially many listing
> operations, this could cause locking contention.
> I think we should add a call {{EncryptionZoneManager#hasEncryptionZone()}} to
> return whether the system has any EZ. If no EZ at all,
> {{getFileEncryptionInfo}} should return null without {{readLock}}.
> If {{hasEncryptionZone}} is only used in the above scenario, maybe itself
> doesn't need a {{readLock}} -- if the system doesn't have any EZ when
> {{getFileEncryptionInfo}} is called on a path, it means the path cannot be
> encrypted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10488) WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating files/directories without specifying permissions
[
https://issues.apache.org/jira/browse/HDFS-10488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316919#comment-15316919
]
John Zhuge commented on HDFS-10488:
---
[~wchevreuil] Shouldn't you take the ownership of this jira?
> WebHDFS CREATE and MKDIRS does not follow same rules as DFS CLI when creating
> files/directories without specifying permissions
> --
>
> Key: HDFS-10488
> URL: https://issues.apache.org/jira/browse/HDFS-10488
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.6.0
>Reporter: Wellington Chevreuil
>Priority: Minor
> Attachments: HDFS-10488.002.patch, HDFS-10488.003.patch,
> HDFS-10488.patch
>
>
> WebHDFS methods for creating file/directories are always creating it with 755
> permissions as default, even ignoring any configured
> *fs.permissions.umask-mode* in the case of directories.
> Dfs CLI, however, applies the configured umask to 777 permission for
> directories, or 666 permission for files.
> Example below shows the different behaviour when creating directory via CLI
> and WebHDFS:
> {noformat}
> 1) Creating a directory under '/test/' as 'test-user'. Configured
> fs.permissions.umask-mode is 000:
> $ sudo -u test-user hdfs dfs -mkdir /test/test-user1
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user1
> # file: /test/test-user1
> # owner: test-user
> # group: supergroup
> user::rwx
> group::rwx
> other::rwx
> 4) Doing the same via WebHDFS does not get the proper ACLs:
> $ curl -i -X PUT
> "http://namenode-host:50070/webhdfs/v1/test/test-user2?user.name=test-user=MKDIRS;
>
> $ sudo -u test-user hdfs dfs -getfacl /test/test-user2
> # file: /test/test-user2
> # owner: test-user
> # group: supergroup
> user::rwx
> group::r-x
> other::r-x
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316915#comment-15316915
]
Hadoop QA commented on HDFS-10375:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} HDFS-10375 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12808433/HDFS-10375.001.patch |
| JIRA Issue | HDFS-10375 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/15663/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Status: Patch Available (was: Open)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10375:
--
Comment: was deleted
(was: +1(non-binding) after reverting the change in {{ValueQueue.java}}. File a
new jira to fix that typo.)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Status: Open (was: Patch Available)
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundant TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HDFS-10375:
--
Summary: Remove redundant TestMiniDFSCluster.testDualClusters (was: Remove
redundent TestMiniDFSCluster.testDualClusters)
> Remove redundant TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316910#comment-15316910
]
John Zhuge commented on HDFS-10375:
---
+1(non-binding) after reverting the change in {{ValueQueue.java}}. File a new
jira to fix that typo.
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316911#comment-15316911
]
John Zhuge commented on HDFS-10375:
---
+1(non-binding) after reverting the change in {{ValueQueue.java}}. File a new
jira to fix that typo.
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[ https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316907#comment-15316907 ] James Clampffer commented on HDFS-9890: --- Committed to HDFS-8707. Thanks for the patch Xiaowei! > libhdfs++: Add test suite to simulate network issues > > > Key: HDFS-9890 > URL: https://issues.apache.org/jira/browse/HDFS-9890 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-9890.HDFS-8707.000.patch, > HDFS-9890.HDFS-8707.001.patch, HDFS-9890.HDFS-8707.002.patch, > HDFS-9890.HDFS-8707.003.patch, HDFS-9890.HDFS-8707.004.patch, > HDFS-9890.HDFS-8707.005.patch, HDFS-9890.HDFS-8707.006.patch, > HDFS-9890.HDFS-8707.007.patch, hs_err_pid26832.log, hs_err_pid4944.log > > > I propose adding a test suite to simulate various network issues/failures in > order to get good test coverage on some of the retry paths that aren't easy > to hit in mock unit tests. > At the moment the only things that hit the retry paths are the gmock unit > tests. The gmock are only as good as their mock implementations which do a > great job of simulating protocol correctness but not more complex > interactions. They also can't really simulate the types of lock contention > and subtle memory stomps that show up while doing hundreds or thousands of > concurrent reads. We should add a new minidfscluster test that focuses on > heavy read/seek load and then randomly convert error codes returned by > network functions into errors. > List of things to simulate(while heavily loaded), roughly in order of how > badly I think they need to be tested at the moment: > -Rpc connection disconnect > -Rpc connection slowed down enough to cause a timeout and trigger retry > -DN connection disconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9890) libhdfs++: Add test suite to simulate network issues
[ https://issues.apache.org/jira/browse/HDFS-9890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9890: -- Resolution: Fixed Status: Resolved (was: Patch Available) > libhdfs++: Add test suite to simulate network issues > > > Key: HDFS-9890 > URL: https://issues.apache.org/jira/browse/HDFS-9890 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Xiaowei Zhu > Attachments: HDFS-9890.HDFS-8707.000.patch, > HDFS-9890.HDFS-8707.001.patch, HDFS-9890.HDFS-8707.002.patch, > HDFS-9890.HDFS-8707.003.patch, HDFS-9890.HDFS-8707.004.patch, > HDFS-9890.HDFS-8707.005.patch, HDFS-9890.HDFS-8707.006.patch, > HDFS-9890.HDFS-8707.007.patch, hs_err_pid26832.log, hs_err_pid4944.log > > > I propose adding a test suite to simulate various network issues/failures in > order to get good test coverage on some of the retry paths that aren't easy > to hit in mock unit tests. > At the moment the only things that hit the retry paths are the gmock unit > tests. The gmock are only as good as their mock implementations which do a > great job of simulating protocol correctness but not more complex > interactions. They also can't really simulate the types of lock contention > and subtle memory stomps that show up while doing hundreds or thousands of > concurrent reads. We should add a new minidfscluster test that focuses on > heavy read/seek load and then randomly convert error codes returned by > network functions into errors. > List of things to simulate(while heavily loaded), roughly in order of how > badly I think they need to be tested at the moment: > -Rpc connection disconnect > -Rpc connection slowed down enough to cause a timeout and trigger retry > -DN connection disconnect -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Attachment: HDFS-10375.001.patch
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
> Attachments: HDFS-10375.001.patch
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou updated HDFS-10375:
--
Fix Version/s: 2.8.0
Release Note: Remove redundent TestMiniDFSCluster.testDualClusters to save
time.
Status: Patch Available (was: Open)
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
> Fix For: 2.8.0
>
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10468) HDFS read ends up ignoring an interrupt
[
https://issues.apache.org/jira/browse/HDFS-10468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15316896#comment-15316896
]
Masatake Iwasaki commented on HDFS-10468:
-
bq. But considering the current complexity of the DFSInputStream code, I do not
plan to fix all of them in this jira.
I agree to fix this incrementally.
While testing the 002, same exception was logged multiple times because the
exception thrown by {{DFSInputStream#checkInterrupted}} is passed to
{{DFSInputStream#checkInterrupted}} again in outer scope. It might to be better
to use debug level for logging in {{DFSInputStream#checkInterrupted}}.
{noformat}
2016-06-07 01:51:13,191 [Thread-84] WARN impl.BlockReaderFactory
(BlockReaderFactory.java:getRemoteBlockReaderFromTcp(758)) - I/O error
constructing remote block reader.
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/127.0.0.1:37109
remote=/127.0.0.1:34420]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
...(snip)
2016-06-07 01:51:13,194 [Thread-84] WARN hdfs.DFSClient
(DFSInputStream.java:checkInterrupted(693)) - The reading thread has been
interrupted.
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/127.0.0.1:37109
remote=/127.0.0.1:34420]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
...(snip)
2016-06-07 01:51:13,194 [Thread-84] WARN hdfs.DFSClient
(DFSInputStream.java:checkInterrupted(693)) - The reading thread has been
interrupted.
java.io.InterruptedIOException: Interrupted while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/127.0.0.1:37109
remote=/127.0.0.1:34420]. 6 millis timeout left.
at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:342)
...(snip)
{noformat}
I'm +1 if this is addressed.
> HDFS read ends up ignoring an interrupt
> ---
>
> Key: HDFS-10468
> URL: https://issues.apache.org/jira/browse/HDFS-10468
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Jing Zhao
> Attachments: HDFS-10468.000.patch, HDFS-10468.001.patch,
> HDFS-10468.002.patch, log
>
>
> If an interrupt comes in during an HDFS read - it looks like HDFS ends up
> ignoring it (handling it), and retries the read after an interval.
> An interrupt should result in the read being cancelled, with an
> InterruptedException being thrown.
> Similarly - if an HDFS op is started with the interrupt status on the thread
> set, an InterruptedException should be thrown.
> cc [~jingzhao]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-10375) Remove redundent TestMiniDFSCluster.testDualClusters
[
https://issues.apache.org/jira/browse/HDFS-10375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Zhou reassigned HDFS-10375:
-
Assignee: Jiayi Zhou
> Remove redundent TestMiniDFSCluster.testDualClusters
>
>
> Key: HDFS-10375
> URL: https://issues.apache.org/jira/browse/HDFS-10375
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: Jiayi Zhou
>Priority: Trivial
> Labels: newbie
>
> Unit test {{TestMiniDFSCluster.testDualClusters}} is redundant because
> {{testClusterWithoutSystemProperties}} already proves
> {{cluster.getDataDirectory() == getProp(HDFS_MINIDFS_BASEDIR) + "/data"}}.
> This unit test sets HDFS_MINIDFS_BASEDIR to 2 different values and brings up
> 2 clusters, of course they will have different data directory.
> Remove it to save the time to bring up 2 mini clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
