[jira] [Commented] (HDFS-13166) [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls
[
https://issues.apache.org/jira/browse/HDFS-13166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379925#comment-16379925
]
genericqa commented on HDFS-13166:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
15s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} HDFS-10285 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 522 unchanged - 0 fixed = 524 total (was 522) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 51s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}150m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}204m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.namenode.TestTruncateQuotaUpdate |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13166 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912374/HDFS-13166-HDFS-10285-02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc xml |
| uname | Linux 5199b66e2907 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revi
[jira] [Created] (HDFS-13203) Ozone: dozone: auto-bootstrap SCM/KSM in containerized environment
Elek, Marton created HDFS-13203: --- Summary: Ozone: dozone: auto-bootstrap SCM/KSM in containerized environment Key: HDFS-13203 URL: https://issues.apache.org/jira/browse/HDFS-13203 Project: Hadoop HDFS Issue Type: Sub-task Components: HDFS-7240 Affects Versions: 3.1.0 Reporter: Elek, Marton Assignee: Elek, Marton Currently to start SCM and KSM in a new environment we need to: 1. Initialize scm (/hdfs scm -init) 2. Start scm 3. hdfs ksm -createObjectStore (connects to the scm, scm should be running) 4. Start ksm The problematic part is step 3.) as in a containerized environment (docker-compose/kubernetes) the timing couldn't be guaranteed. I propose to introduce a new command line flag '--auto-bootstrap' for both KSM and SCM (and maybe for Namenode in the future). It will: 1. Initilize KSM and SCM *if and only if* the data directories are missing 2. In case of KSM retry the initilization in every 10 seconds (in case of SCM is not yet running) in case of connection failure -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13166) [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls
[
https://issues.apache.org/jira/browse/HDFS-13166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-13166:
Attachment: HDFS-13166-HDFS-10285-03.patch
> [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly
> getLiveDatanodeStorageReport() calls
> -
>
> Key: HDFS-13166
> URL: https://issues.apache.org/jira/browse/HDFS-13166
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Attachments: HDFS-13166-HDFS-10285-00.patch,
> HDFS-13166-HDFS-10285-01.patch, HDFS-13166-HDFS-10285-02.patch,
> HDFS-13166-HDFS-10285-03.patch
>
>
> Presently {{#getLiveDatanodeStorageReport()}} is fetched for every file and
> does the computation. This Jira sub-task is to discuss and implement a cache
> mechanism which in turn reduces the number of function calls. Also, could
> define a configurable refresh interval and periodically refresh the DN cache
> by fetching latest {{#getLiveDatanodeStorageReport}} on this interval.
> Following comments taken from HDFS-10285,
> [here|https://issues.apache.org/jira/browse/HDFS-10285?focusedCommentId=16347472&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16347472]
> Comment-7)
> {quote}Adding getDatanodeStorageReport is concerning.
> getDatanodeListForReport is already a very bad method that should be avoided
> for anything but jmx – even then it’s a concern. I eliminated calls to it
> years ago. All it takes is a nscd/dns hiccup and you’re left holding the fsn
> lock for an excessive length of time. Beyond that, the response is going to
> be pretty large and tagging all the storage reports is not going to be cheap.
> verifyTargetDatanodeHasSpaceForScheduling does it really need the namesystem
> lock? Can’t DatanodeDescriptor#chooseStorage4Block synchronize on its
> storageMap?
> Appears to be calling getLiveDatanodeStorageReport for every file. As
> mentioned earlier, this is NOT cheap. The SPS should be able to operate on a
> fuzzy/cached state of the world. Then it gets another datanode report to
> determine the number of live nodes to decide if it should sleep before
> processing the next path. The number of nodes from the prior cached view of
> the world should suffice.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13196) Ozone: dozone: make example docker-compose files version independent
[
https://issues.apache.org/jira/browse/HDFS-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1638#comment-1638
]
Elek, Marton commented on HDFS-13196:
-
[~anu] Thank you for the testing:
The problem could be a timing issue (which is introduced earlier and
independent from the versioning problem). The SCM should be running when ksm is
initialized.
I created and issue to handle this (HDFS-13203) and also updated the base image
(HADOOP-15083) with a workaround (it will be removed after HDFS-13203 but it's
a quick workaround as of now):
{code}
if [ -n "$ENSURE_KSM_INITIALIZED" ]; then
if [ ! -f "$ENSURE_KSM_INITIALIZED" ]; then
+ #To make sure SCM is running in dockerized environment we will sleep
+ # Could be removed after HDFS-13203
+ echo "Waiting 15 seconds for SCM startup"
+ sleep 15
/opt/hadoop/bin/hdfs ksm -createObjectStore
fi
fi
{code}
I also uploaded the latest proposed hadoop-runner version to the dockerhub with
the name elek/hadoop-runner.
Could you please try this patch again after:
{code}
docker-compose down
docker-compose pull
{down}
I retested it after a clean run and it worked for me:
> Ozone: dozone: make example docker-compose files version independent
>
>
> Key: HDFS-13196
> URL: https://issues.apache.org/jira/browse/HDFS-13196
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13196-HDFS-7240.001.patch
>
>
> The current version of the docker-compose files at dev-support/compose/ are
> version dependent as the path of the dist folder should be added to the
> docker-compose file.
> In this patch I propose to add these compose files to the dist project and
> replace the version number in the docker-compose files during the build.
> Please note: it doesn't mean that the docker-compose files are part of the
> distribution. The compose files are copied to hadoopd-dist/target/compose
> which is not part of the hadoop distribution.
> This patch is aligned with HADOOP-15257 which uses exactly the same approach
> to provide example docker-compose file for hdfs cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13190) Document WebHDFS support for snasphot diff
[ https://issues.apache.org/jira/browse/HDFS-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13190: --- Attachment: HDFS-13190.001.patch > Document WebHDFS support for snasphot diff > -- > > Key: HDFS-13190 > URL: https://issues.apache.org/jira/browse/HDFS-13190 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, webhdfs >Reporter: Xiaoyu Yao >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13190.001.patch > > > This ticket is opened to document the WebHDFS: Add support for snasphot diff > from HDFS-13052 in WebHDFS.md. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13190) Document WebHDFS support for snasphot diff
[ https://issues.apache.org/jira/browse/HDFS-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13190: --- Status: Patch Available (was: Open) > Document WebHDFS support for snasphot diff > -- > > Key: HDFS-13190 > URL: https://issues.apache.org/jira/browse/HDFS-13190 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, webhdfs >Reporter: Xiaoyu Yao >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13190.001.patch > > > This ticket is opened to document the WebHDFS: Add support for snasphot diff > from HDFS-13052 in WebHDFS.md. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13141) WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380045#comment-16380045 ] Lokesh Jain commented on HDFS-13141: v2 patch fixes checkstyle issues. > WebHDFS: Add support for getting snasphottable directory list > - > > Key: HDFS-13141 > URL: https://issues.apache.org/jira/browse/HDFS-13141 > Project: Hadoop HDFS > Issue Type: Task > Components: webhdfs >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13141.001.patch, HDFS-13141.002.patch > > > This Jira aims to implement get snapshottable directory list operation for > webHdfs filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13141) WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13141: --- Attachment: HDFS-13141.002.patch > WebHDFS: Add support for getting snasphottable directory list > - > > Key: HDFS-13141 > URL: https://issues.apache.org/jira/browse/HDFS-13141 > Project: Hadoop HDFS > Issue Type: Task > Components: webhdfs >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13141.001.patch, HDFS-13141.002.patch > > > This Jira aims to implement get snapshottable directory list operation for > webHdfs filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[ https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-13102: --- Attachment: HDFS-13102.008.patch > Implement SnapshotSkipList class to store Multi level DirectoryDiffs > > > Key: HDFS-13102 > URL: https://issues.apache.org/jira/browse/HDFS-13102 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13102.001.patch, HDFS-13102.002.patch, > HDFS-13102.003.patch, HDFS-13102.004.patch, HDFS-13102.005.patch, > HDFS-13102.006.patch, HDFS-13102.007.patch, HDFS-13102.008.patch > > > HDFS-11225 explains an issue where deletion of older snapshots can take a > very long time in case the no of snapshot diffs is quite large for > directories. For any directory under a snapshot, to construct the children > list , it needs to combine all the diffs from that particular snapshot to the > last snapshotDiff record and reverseApply to the current children list of the > directory on live fs. This can take a significant time if the no of snapshot > diffs are quite large and changes per diff is significant. > This Jira proposes to store the Directory diffs in a SnapshotSkip list, where > we store multi level DirectoryDiffs. At each level, the Directory Diff will > be cumulative diff of k snapshot diffs, > where k is the level of a node in the list. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[ https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380049#comment-16380049 ] Shashikant Banerjee commented on HDFS-13102: Thanks [~szetszwo], for the review. patch v8 addresses your review comments. Please have a look. > Implement SnapshotSkipList class to store Multi level DirectoryDiffs > > > Key: HDFS-13102 > URL: https://issues.apache.org/jira/browse/HDFS-13102 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13102.001.patch, HDFS-13102.002.patch, > HDFS-13102.003.patch, HDFS-13102.004.patch, HDFS-13102.005.patch, > HDFS-13102.006.patch, HDFS-13102.007.patch, HDFS-13102.008.patch > > > HDFS-11225 explains an issue where deletion of older snapshots can take a > very long time in case the no of snapshot diffs is quite large for > directories. For any directory under a snapshot, to construct the children > list , it needs to combine all the diffs from that particular snapshot to the > last snapshotDiff record and reverseApply to the current children list of the > directory on live fs. This can take a significant time if the no of snapshot > diffs are quite large and changes per diff is significant. > This Jira proposes to store the Directory diffs in a SnapshotSkip list, where > we store multi level DirectoryDiffs. At each level, the Directory Diff will > be cumulative diff of k snapshot diffs, > where k is the level of a node in the list. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12975) Changes to the NameNode to support reads from standby
[
https://issues.apache.org/jira/browse/HDFS-12975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380068#comment-16380068
]
genericqa commented on HDFS-12975:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12975 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912394/HDFS-12975.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0f2264fa1897 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a9c14b1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23237/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/Pr
[jira] [Commented] (HDFS-13190) Document WebHDFS support for snasphot diff
[
https://issues.apache.org/jira/browse/HDFS-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380085#comment-16380085
]
genericqa commented on HDFS-13190:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
26m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 22s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13190 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912412/HDFS-13190.001.patch |
| Optional Tests | asflicense mvnsite |
| uname | Linux f0a8c3b95464 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a9c14b1 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 441 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23241/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Document WebHDFS support for snasphot diff
> --
>
> Key: HDFS-13190
> URL: https://issues.apache.org/jira/browse/HDFS-13190
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, webhdfs
>Reporter: Xiaoyu Yao
>Assignee: Lokesh Jain
>Priority: Major
> Attachments: HDFS-13190.001.patch
>
>
> This ticket is opened to document the WebHDFS: Add support for snasphot diff
> from HDFS-13052 in WebHDFS.md.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13202) Fix javadoc in HAUtil and small refactoring
[
https://issues.apache.org/jira/browse/HDFS-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380086#comment-16380086
]
genericqa commented on HDFS-13202:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 56s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.TestDFSClientRetries |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13202 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912395/HDFS-13202.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 32fd6dff899d 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a9c14b1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23238/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23238/te
[jira] [Created] (HDFS-13204) RBF: optimize name service safe mode icon
liuhongtong created HDFS-13204: -- Summary: RBF: optimize name service safe mode icon Key: HDFS-13204 URL: https://issues.apache.org/jira/browse/HDFS-13204 Project: Hadoop HDFS Issue Type: Wish Reporter: liuhongtong Attachments: image-2018-02-28-17-46-35-127.png, image-2018-02-28-17-47-44-156.png, image-2018-02-28-17-51-33-913.png In federationhealth webpage，the safe mode icon in the picture below may induce users the name service is maintaining. !image-2018-02-28-17-51-33-913.png! In fact, if the name service is in safe mode, users can't do most of operations. So I think the icon below may be more reasonable. !image-2018-02-28-17-46-35-127.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Issue Type: Sub-task (was: Wish) Parent: HDFS-12615 > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: image-2018-02-28-17-46-35-127.png, > image-2018-02-28-17-47-44-156.png, image-2018-02-28-17-51-33-913.png > > > In federationhealth webpage，the safe mode icon in the picture below may > induce users the name service is maintaining. > !image-2018-02-28-17-51-33-913.png! > In fact, if the name service is in safe mode, users can't do most of > operations. So I think the icon below may be more reasonable. > !image-2018-02-28-17-46-35-127.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Attachment: image-2018-02-28-18-23-23-909.png Description: In federation health webpage, the safe mode icons of Subclusters and Routers are inconsistent. The safe mode icon of Subclusters may induce users the name service is maintaining. !image-2018-02-28-18-22-51-975.png! The safe mode icon of Routers: !image-2018-02-28-18-23-23-909.png! In fact, if the name service is in safe mode, users can't do writing related operations. So I think the icon below may be more reasonable. !image-2018-02-28-17-46-35-127.png! was: In federationhealth webpage，the safe mode icon in the picture below may induce users the name service is maintaining. !image-2018-02-28-17-51-33-913.png! In fact, if the name service is in safe mode, users can't do most of operations. So I think the icon below may be more reasonable. !image-2018-02-28-17-46-35-127.png! > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: image-2018-02-28-17-46-35-127.png, > image-2018-02-28-17-47-44-156.png, image-2018-02-28-17-51-33-913.png, > image-2018-02-28-18-23-23-909.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-22-51-975.png! > The safe mode icon of Routers: > !image-2018-02-28-18-23-23-909.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the icon below may be more reasonable. > !image-2018-02-28-17-46-35-127.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Attachment: image-2018-02-28-18-33-09-972.png image-2018-02-28-18-33-47-661.png image-2018-02-28-18-35-35-708.png Description: In federation health webpage, the safe mode icons of Subclusters and Routers are inconsistent. The safe mode icon of Subclusters may induce users the name service is maintaining. !image-2018-02-28-18-33-09-972.png! The safe mode icon of Routers: !image-2018-02-28-18-33-47-661.png! In fact, if the name service is in safe mode, users can't do writing related operations. So I think the icon below may be more reasonable. !image-2018-02-28-18-35-35-708.png! was: In federation health webpage, the safe mode icons of Subclusters and Routers are inconsistent. The safe mode icon of Subclusters may induce users the name service is maintaining. !image-2018-02-28-18-22-51-975.png! The safe mode icon of Routers: !image-2018-02-28-18-23-23-909.png! In fact, if the name service is in safe mode, users can't do writing related operations. So I think the icon below may be more reasonable. !image-2018-02-28-17-46-35-127.png! > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: image-2018-02-28-18-23-23-909.png, > image-2018-02-28-18-33-09-972.png, image-2018-02-28-18-33-47-661.png, > image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the icon below may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Attachment: (was: image-2018-02-28-17-51-33-913.png) > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: image-2018-02-28-18-23-23-909.png, > image-2018-02-28-18-33-09-972.png, image-2018-02-28-18-33-47-661.png, > image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the icon below may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Attachment: (was: image-2018-02-28-17-46-35-127.png) > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: image-2018-02-28-18-23-23-909.png, > image-2018-02-28-18-33-09-972.png, image-2018-02-28-18-33-47-661.png, > image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the icon below may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Attachment: (was: image-2018-02-28-17-47-44-156.png) > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: image-2018-02-28-18-23-23-909.png, > image-2018-02-28-18-33-09-972.png, image-2018-02-28-18-33-47-661.png, > image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the icon below may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Attachment: (was: image-2018-02-28-18-23-23-909.png) > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the icon below may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Description: In federation health webpage, the safe mode icons of Subclusters and Routers are inconsistent. The safe mode icon of Subclusters may induce users the name service is maintaining. !image-2018-02-28-18-33-09-972.png! The safe mode icon of Routers: !image-2018-02-28-18-33-47-661.png! In fact, if the name service is in safe mode, users can't do writing related operations. So I think the safe mode icon in Subclusters should be modify, which may be more reasonable. !image-2018-02-28-18-35-35-708.png! was: In federation health webpage, the safe mode icons of Subclusters and Routers are inconsistent. The safe mode icon of Subclusters may induce users the name service is maintaining. !image-2018-02-28-18-33-09-972.png! The safe mode icon of Routers: !image-2018-02-28-18-33-47-661.png! In fact, if the name service is in safe mode, users can't do writing related operations. So I think the icon below may be more reasonable. !image-2018-02-28-18-35-35-708.png! > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modify, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Description: In federation health webpage, the safe mode icons of Subclusters and Routers are inconsistent. The safe mode icon of Subclusters may induce users the name service is maintaining. !image-2018-02-28-18-33-09-972.png! The safe mode icon of Routers: !image-2018-02-28-18-33-47-661.png! In fact, if the name service is in safe mode, users can't do writing related operations. So I think the safe mode icon in Subclusters should be modified, which may be more reasonable. !image-2018-02-28-18-35-35-708.png! was: In federation health webpage, the safe mode icons of Subclusters and Routers are inconsistent. The safe mode icon of Subclusters may induce users the name service is maintaining. !image-2018-02-28-18-33-09-972.png! The safe mode icon of Routers: !image-2018-02-28-18-33-47-661.png! In fact, if the name service is in safe mode, users can't do writing related operations. So I think the safe mode icon in Subclusters should be modify, which may be more reasonable. !image-2018-02-28-18-35-35-708.png! > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Attachment: HDFS-13204.001.patch > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuhongtong updated HDFS-13204: --- Status: Patch Available (was: Open) > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380139#comment-16380139
]
He Xiaoqiao commented on HDFS-13183:
[~xkrogen]
Thanks for your detailed comments, and sorry for slow response.
{quote}we should not remove checkOperation altogether, but rather it should be
OperationCategory.UNCHECKED. We should have this feature be opt-in.{quote}
It is good suggestions, and I will update this issue following your advice.
{quote}Additionally the exception that should be thrown to purposefully trigger
failover for a client is currently a StandbyException, not a generic
IOException. {quote}
will client not failover immediately when it meet IOException even if
{{NamenodeProtocol#getBlocks}} is idempotent? maybe i don't understand
correctly, if that please correct me.
Thanks again for [~xkrogen] pushing this.
> Standby NameNode process getBlocks request to reduce Active load
>
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
>Affects Versions: 2.7.5, 3.1.0, 2.9.1, 2.8.4, 3.0.2
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380156#comment-16380156 ] tartarus commented on HDFS-13204: - LGTM > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: optimize name service safe mode icon
[
https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380162#comment-16380162
]
genericqa commented on HDFS-13204:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
24m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13204 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912430/HDFS-13204.001.patch |
| Optional Tests | asflicense shadedclient |
| uname | Linux f80429182efb 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7af4f34 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 441 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23244/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RBF: optimize name service safe mode icon
> -
>
> Key: HDFS-13204
> URL: https://issues.apache.org/jira/browse/HDFS-13204
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Priority: Minor
> Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png,
> image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png
>
>
> In federation health webpage, the safe mode icons of Subclusters and Routers
> are inconsistent.
> The safe mode icon of Subclusters may induce users the name service is
> maintaining.
> !image-2018-02-28-18-33-09-972.png!
> The safe mode icon of Routers:
> !image-2018-02-28-18-33-47-661.png!
> In fact, if the name service is in safe mode, users can't do writing related
> operations. So I think the safe mode icon in Subclusters should be modified,
> which may be more reasonable.
> !image-2018-02-28-18-35-35-708.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12749) DN may not send block report to NN after NN restart
[
https://issues.apache.org/jira/browse/HDFS-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380179#comment-16380179
]
He Xiaoqiao commented on HDFS-12749:
ping [~xkrogen],[~kihwal] any suggestions or feedback for this issue?
> DN may not send block report to NN after NN restart
> ---
>
> Key: HDFS-12749
> URL: https://issues.apache.org/jira/browse/HDFS-12749
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1, 2.8.3, 2.7.5, 3.0.0, 2.9.1
>Reporter: TanYuxin
>Priority: Major
> Attachments: HDFS-12749-branch-2.7.002.patch,
> HDFS-12749-trunk.003.patch, HDFS-12749.001.patch
>
>
> Now our cluster have thousands of DN, millions of files and blocks. When NN
> restart, NN's load is very high.
> After NN restart，DN will call BPServiceActor#reRegister method to register.
> But register RPC will get a IOException since NN is busy dealing with Block
> Report. The exception is caught at BPServiceActor#processCommand.
> Next is the caught IOException:
> {code:java}
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing
> datanode Command
> java.io.IOException: Failed on local exception: java.io.IOException:
> java.net.SocketTimeoutException: 6 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/DataNode_IP:Port remote=NameNode_Host/IP:Port]; Host Details : local
> host is: "DataNode_Host/Datanode_IP"; destination host is:
> "NameNode_Host":Port;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
> at org.apache.hadoop.ipc.Client.call(Client.java:1474)
> at org.apache.hadoop.ipc.Client.call(Client.java:1407)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy13.registerDatanode(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:126)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:793)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reRegister(BPServiceActor.java:926)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:604)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:898)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:711)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:864)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The un-catched IOException breaks BPServiceActor#register, and the Block
> Report can not be sent immediately.
> {code}
> /**
>* Register one bp with the corresponding NameNode
>*
>* The bpDatanode needs to register with the namenode on startup in order
>* 1) to report which storage it is serving now and
>* 2) to receive a registrationID
>*
>* issued by the namenode to recognize registered datanodes.
>*
>* @param nsInfo current NamespaceInfo
>* @see FSNamesystem#registerDatanode(DatanodeRegistration)
>* @throws IOException
>*/
> void register(NamespaceInfo nsInfo) throws IOException {
> // The handshake() phase loaded the block pool storage
> // off disk - so update the bpRegistration object from that info
> DatanodeRegistration newBpRegistration = bpos.createRegistration();
> LOG.info(this + " beginning handshake with NN");
> while (shouldRun()) {
> try {
> // Use returned registration from namenode with updated fields
> newBpRegistration = bpNamenode.registerDatanode(newBpRegistration);
> newBpRegistration.setNamespaceInfo(nsInfo);
> bpRegistration = newBpRegistration;
> break;
> } catch(EOFException e) { // namenode might have just restarted
> LOG.info("Problem connecting to server: " + nnAddr + " :"
> + e.getLocalizedMessage());
> sleepAndLogInterrupts(1000, "connecting to server");
> } catch(SocketTimeoutException e) { // namenode is busy
> LOG.info("Problem connecting to server: " + nnAddr);
> sleepAndLogInterrupts(1000, "connecting to server");
> }
> }
>
> LOG.info("Block pool " + this + " successfully registered with NN");
> bpos.registrationSucceeded(this, bpRegistration);
> // random short delay - helps scatter the BR from all DNs
> scheduler.scheduleBlockReport(dnConf.initialBlockReportDelay);
> }
> {code}
> But NameNode has processed registerDatanode successfully, so
[jira] [Commented] (HDFS-13166) [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls
[
https://issues.apache.org/jira/browse/HDFS-13166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380222#comment-16380222
]
genericqa commented on HDFS-13166:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-10285 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
26s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} HDFS-10285 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} HDFS-10285 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}147m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}203m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.namenode.TestTruncateQuotaUpdate |
| | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13166 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912408/HDFS-13166-HDFS-10285-03.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc xml |
| uname | Linux 8c59a654dd3d 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| gi
[jira] [Commented] (HDFS-13141) WebHDFS: Add support for getting snasphottable directory list
[
https://issues.apache.org/jira/browse/HDFS-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380252#comment-16380252
]
genericqa commented on HDFS-13141:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
241 unchanged - 3 fixed = 242 total (was 244) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}124m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}181m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13141 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912413/HDFS-13141.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9ab608fb06b0 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64
[jira] [Commented] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[
https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380254#comment-16380254
]
genericqa commented on HDFS-13102:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 14 unchanged - 0 fixed = 18 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 44s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}127m 22s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13102 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912414/HDFS-13102.008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2dbe88f61a8c 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a9c14b1 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23243/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23243/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/2324
[jira] [Updated] (HDFS-13171) Handle Deletion of nodes in SnasphotSkipList
[ https://issues.apache.org/jira/browse/HDFS-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-13171: --- Attachment: HDFS-13171.000.patch > Handle Deletion of nodes in SnasphotSkipList > > > Key: HDFS-13171 > URL: https://issues.apache.org/jira/browse/HDFS-13171 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13171.000.patch > > > This Jira will handle deletion of skipListNodes from DirectoryDiffList . If a > node has multiple levels, the list needs to be balanced .If the node is uni > level, no balancing of the list is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13171) Handle Deletion of nodes in SnasphotSkipList
[ https://issues.apache.org/jira/browse/HDFS-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380363#comment-16380363 ] Shashikant Banerjee commented on HDFS-13171: Patch v0 is dependent on HDFS-13102. Not submitting it for now. > Handle Deletion of nodes in SnasphotSkipList > > > Key: HDFS-13171 > URL: https://issues.apache.org/jira/browse/HDFS-13171 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13171.000.patch > > > This Jira will handle deletion of skipListNodes from DirectoryDiffList . If a > node has multiple levels, the list needs to be balanced .If the node is uni > level, no balancing of the list is required. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13205) Incorrect path is passed to checkPermission during authorization of file under a snapshot (specifically under a subdir) after original subdir is deleted
Raghavender Rao Guruvannagari created HDFS-13205:
Summary: Incorrect path is passed to checkPermission during
authorization of file under a snapshot (specifically under a subdir) after
original subdir is deleted
Key: HDFS-13205
URL: https://issues.apache.org/jira/browse/HDFS-13205
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 2.7.4
Reporter: Raghavender Rao Guruvannagari
Steps to reproduce the issue.
+As 'hdfs' superuser+
-- Create a folder (/hdptest/test) with 700 permissions and (
/hdptest/test/mydir) with 755.
--HDFS Ranger policy is defined with RWX for user "test" on /hdptest/test/
recursively.
--Allow snapshot on the directory /hdptest/test/mydir:
{code:java}
#su - test
[test@node1 ~]$ hdfs dfs -ls /hdptest/test/mydir
[test@node1 ~]$ hdfs dfs -mkdir /hdptest/test/mydir/test
[test@node1 ~]$ hdfs dfs -put /etc/passwd /hdptest/test/mydir/test
[test@node1 ~]$ hdfs lsSnapshottableDir
drwxr-xr-x 0 test hdfs 0 2018-01-25 14:22 1 65536 /hdptest/test/mydir
{code}
-->Create Snapshot :
{code:java}
[test@node1 ~]$ hdfs dfs -createSnapshot /hdptest/test/mydir
Created snapshot /hdptest/test/mydir/.snapshot/s20180125-135430.953
{code}
-->Verifying that snapshot directory has the current files from directory and
verify the file is accessible .snapshot path:
{code:java}
[test@node1 ~]$ hdfs dfs -ls -R
/hdptest/test/mydir/.snapshot/s20180125-135430.953
drwxr-xr-x - test hdfs 0 2018-01-25 13:53
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test
-rw-r--r-- 3 test hdfs 3227 2018-01-25 13:53
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd
[test@node1 ~]$ hdfs dfs -cat
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd | tail
livytest:x:1015:496::/home/livytest:/bin/bash
ehdpzepp:x:1016:496::/home/ehdpzepp:/bin/bash
zepptest:x:1017:496::/home/zepptest:/bin/bash
{code}
-->Remove the file from main directory and verified that file is still
accessible:
{code:java}
[test@node1 ~]$ hdfs dfs -rm /hdptest/test/mydir/test/passwd
18/01/25 13:55:06 INFO fs.TrashPolicyDefault: Moved:
'hdfs://rangerSME/hdptest/test/mydir/test/passwd' to trash at:
hdfs://rangerSME/user/test/.Trash/Current/hdptest/test/mydir/test/passwd
[test@node1 ~]$ hdfs dfs -cat
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd | tail
livytest:x:1015:496::/home/livytest:/bin/bash
{code}
-->Remove the parent directory of the file which was deleted, now accessing the
same file under .snapshot dir fails with permission denied error
{code:java}
[test@node1 ~]$ hdfs dfs -rm -r /hdptest/test/mydir/test
18/01/25 13:55:25 INFO fs.TrashPolicyDefault: Moved:
'hdfs://rangerSME/hdptest/test/mydir/test' to trash at:
hdfs://rangerSME/user/test/.Trash/Current/hdptest/test/mydir/test1516888525269
[test@node1 ~]$ hdfs dfs -cat
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd | tail
cat: Permission denied: user=test, access=EXECUTE,
inode="/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd":hdfs:hdfs:drwxr-x---
{code}
Ranger policies are not honored in this case for .snapshot directories/files
after main directory is deleted under snapshotable directory.
Workaround is to provide execute permission at HDFS level for the parent folder
{code:java}
#su - hdfs
#hdfs dfs -chmod 701 /hdptest/test
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13205) Incorrect path is passed to checkPermission during authorization of file under a snapshot (specifically under a subdir) after original subdir is deleted
[
https://issues.apache.org/jira/browse/HDFS-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghavender Rao Guruvannagari updated HDFS-13205:
-
Description:
Steps to reproduce the issue.
+As 'hdfs' superuser+
– Create a folder (/hdptest/test) with 700 permissions and (
/hdptest/test/mydir) with 755.
--HDFS Ranger policy is defined with RWX for user "test" on /hdptest/test/
recursively.
--Allow snapshot on the directory /hdptest/test/mydir:
{code:java}
#su - test
[test@node1 ~]$ hdfs dfs -ls /hdptest/test/mydir
[test@node1 ~]$ hdfs dfs -mkdir /hdptest/test/mydir/test
[test@node1 ~]$ hdfs dfs -put /etc/passwd /hdptest/test/mydir/test
[test@node1 ~]$ hdfs lsSnapshottableDir
drwxr-xr-x 0 test hdfs 0 2018-01-25 14:22 1 65536 /hdptest/test/mydir
{code}
-->Create Snapshot
{code:java}
[test@node1 ~]$ hdfs dfs -createSnapshot /hdptest/test/mydir
Created snapshot /hdptest/test/mydir/.snapshot/s20180125-135430.953
{code}
-->Verifying that snapshot directory has the current files from directory and
verify the file is accessible .snapshot path:
{code:java}
[test@node1 ~]$ hdfs dfs -ls -R
/hdptest/test/mydir/.snapshot/s20180125-135430.953
drwxr-xr-x - test hdfs 0 2018-01-25 13:53
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test
-rw-r--r-- 3 test hdfs 3227 2018-01-25 13:53
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd
[test@node1 ~]$ hdfs dfs -cat
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd | tail
livytest:x:1015:496::/home/livytest:/bin/bash
ehdpzepp:x:1016:496::/home/ehdpzepp:/bin/bash
zepptest:x:1017:496::/home/zepptest:/bin/bash
{code}
-->Remove the file from main directory and verified that file is still
accessible:
{code:java}
[test@node1 ~]$ hdfs dfs -rm /hdptest/test/mydir/test/passwd
18/01/25 13:55:06 INFO fs.TrashPolicyDefault: Moved:
'hdfs://rangerSME/hdptest/test/mydir/test/passwd' to trash at:
hdfs://rangerSME/user/test/.Trash/Current/hdptest/test/mydir/test/passwd
[test@node1 ~]$ hdfs dfs -cat
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd | tail
livytest:x:1015:496::/home/livytest:/bin/bash
{code}
-->Remove the parent directory of the file which was deleted, now accessing
the same file under .snapshot dir fails with permission denied error
{code:java}
[test@node1 ~]$ hdfs dfs -rm -r /hdptest/test/mydir/test
18/01/25 13:55:25 INFO fs.TrashPolicyDefault: Moved:
'hdfs://rangerSME/hdptest/test/mydir/test' to trash at:
hdfs://rangerSME/user/test/.Trash/Current/hdptest/test/mydir/test1516888525269
[test@node1 ~]$ hdfs dfs -cat
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd | tail
cat: Permission denied: user=test, access=EXECUTE,
inode="/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd":hdfs:hdfs:drwxr-x---
{code}
Ranger policies are not honored in this case for .snapshot directories/files
after main directory is deleted under snapshotable directory.
Workaround is to provide execute permission at HDFS level for the parent
folder
{code:java}
#su - hdfs
#hdfs dfs -chmod 701 /hdptest/test
{code}
was:
Steps to reproduce the issue.
+As 'hdfs' superuser+
-- Create a folder (/hdptest/test) with 700 permissions and (
/hdptest/test/mydir) with 755.
--HDFS Ranger policy is defined with RWX for user "test" on /hdptest/test/
recursively.
--Allow snapshot on the directory /hdptest/test/mydir:
{code:java}
#su - test
[test@node1 ~]$ hdfs dfs -ls /hdptest/test/mydir
[test@node1 ~]$ hdfs dfs -mkdir /hdptest/test/mydir/test
[test@node1 ~]$ hdfs dfs -put /etc/passwd /hdptest/test/mydir/test
[test@node1 ~]$ hdfs lsSnapshottableDir
drwxr-xr-x 0 test hdfs 0 2018-01-25 14:22 1 65536 /hdptest/test/mydir
{code}
-->Create Snapshot :
{code:java}
[test@node1 ~]$ hdfs dfs -createSnapshot /hdptest/test/mydir
Created snapshot /hdptest/test/mydir/.snapshot/s20180125-135430.953
{code}
-->Verifying that snapshot directory has the current files from directory and
verify the file is accessible .snapshot path:
{code:java}
[test@node1 ~]$ hdfs dfs -ls -R
/hdptest/test/mydir/.snapshot/s20180125-135430.953
drwxr-xr-x - test hdfs 0 2018-01-25 13:53
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test
-rw-r--r-- 3 test hdfs 3227 2018-01-25 13:53
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd
[test@node1 ~]$ hdfs dfs -cat
/hdptest/test/mydir/.snapshot/s20180125-135430.953/test/passwd | tail
livytest:x:1015:496::/home/livytest:/bin/bash
ehdpzepp:x:1016:496::/home/ehdpzepp:/bin/bash
zepptest:x:1017:496::/home/zepptest:/bin/bash
{code}
-->Remove the file from main directory and verified that file is still
accessible:
{code:java}
[test@node1 ~]$ hdfs dfs -rm /hdptest/test/mydir/test/passwd
18/01/25 13:55:06 INFO fs.TrashPolicyDefault: Moved:
'hdfs://rangerSME/hdptest/test/mydir/test/passwd' to trash at:
hdfs://rangerSME/user/
[jira] [Commented] (HDFS-12749) DN may not send block report to NN after NN restart
[
https://issues.apache.org/jira/browse/HDFS-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380482#comment-16380482
]
Erik Krogen commented on HDFS-12749:
I think the v003 patch is fine to resolve this issue. I am not a committer so
cannot help you commit based on my review.
I am not satisfied that we fully understand the over-wrapped exception, but
that is not critical for this bug fix. Maybe let's open a new JIRA to discuss
resolution.
> DN may not send block report to NN after NN restart
> ---
>
> Key: HDFS-12749
> URL: https://issues.apache.org/jira/browse/HDFS-12749
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1, 2.8.3, 2.7.5, 3.0.0, 2.9.1
>Reporter: TanYuxin
>Priority: Major
> Attachments: HDFS-12749-branch-2.7.002.patch,
> HDFS-12749-trunk.003.patch, HDFS-12749.001.patch
>
>
> Now our cluster have thousands of DN, millions of files and blocks. When NN
> restart, NN's load is very high.
> After NN restart，DN will call BPServiceActor#reRegister method to register.
> But register RPC will get a IOException since NN is busy dealing with Block
> Report. The exception is caught at BPServiceActor#processCommand.
> Next is the caught IOException:
> {code:java}
> WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing
> datanode Command
> java.io.IOException: Failed on local exception: java.io.IOException:
> java.net.SocketTimeoutException: 6 millis timeout while waiting for
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected
> local=/DataNode_IP:Port remote=NameNode_Host/IP:Port]; Host Details : local
> host is: "DataNode_Host/Datanode_IP"; destination host is:
> "NameNode_Host":Port;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:773)
> at org.apache.hadoop.ipc.Client.call(Client.java:1474)
> at org.apache.hadoop.ipc.Client.call(Client.java:1407)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy13.registerDatanode(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.registerDatanode(DatanodeProtocolClientSideTranslatorPB.java:126)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:793)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.reRegister(BPServiceActor.java:926)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:604)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:898)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:711)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:864)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The un-catched IOException breaks BPServiceActor#register, and the Block
> Report can not be sent immediately.
> {code}
> /**
>* Register one bp with the corresponding NameNode
>*
>* The bpDatanode needs to register with the namenode on startup in order
>* 1) to report which storage it is serving now and
>* 2) to receive a registrationID
>*
>* issued by the namenode to recognize registered datanodes.
>*
>* @param nsInfo current NamespaceInfo
>* @see FSNamesystem#registerDatanode(DatanodeRegistration)
>* @throws IOException
>*/
> void register(NamespaceInfo nsInfo) throws IOException {
> // The handshake() phase loaded the block pool storage
> // off disk - so update the bpRegistration object from that info
> DatanodeRegistration newBpRegistration = bpos.createRegistration();
> LOG.info(this + " beginning handshake with NN");
> while (shouldRun()) {
> try {
> // Use returned registration from namenode with updated fields
> newBpRegistration = bpNamenode.registerDatanode(newBpRegistration);
> newBpRegistration.setNamespaceInfo(nsInfo);
> bpRegistration = newBpRegistration;
> break;
> } catch(EOFException e) { // namenode might have just restarted
> LOG.info("Problem connecting to server: " + nnAddr + " :"
> + e.getLocalizedMessage());
> sleepAndLogInterrupts(1000, "connecting to server");
> } catch(SocketTimeoutException e) { // namenode is busy
> LOG.info("Problem connecting to server: " + nnAddr);
> sleepAndLogInterrupts(1000, "connecting to server");
> }
> }
>
> LOG.info("Block pool " + this + " successfully registered with NN");
> bpos.registrationSucceeded(this, bpRegis
[jira] [Commented] (HDFS-13183) Standby NameNode process getBlocks request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-13183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380554#comment-16380554
]
Erik Krogen commented on HDFS-13183:
An {{IOException}} thrown on the ANN side will return to the client as a
{{RemoteException}}, so I don't believe it will properly trigger failover.
Regardless, the failover handling of generic {{IOException}} within
{{FailoverOnNetworkExceptionRetry}} is intended for network issues, not
purposeful failover, which is the reason for the existence of
{{StandbyException}}.
> Standby NameNode process getBlocks request to reduce Active load
>
>
> Key: HDFS-13183
> URL: https://issues.apache.org/jira/browse/HDFS-13183
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover, namenode
>Affects Versions: 2.7.5, 3.1.0, 2.9.1, 2.8.4, 3.0.2
>Reporter: He Xiaoqiao
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-13183-trunk.001.patch, HDFS-13183-trunk.002.patch
>
>
> The performance of Active NameNode could be impact when {{Balancer}} requests
> #getBlocks, since query blocks of overly full DNs performance is extremely
> inefficient currently. The main reason is {{NameNodeRpcServer#getBlocks}}
> hold read lock for long time. In extreme case, all handlers of Active
> NameNode RPC server are occupied by one reader
> {{NameNodeRpcServer#getBlocks}} and other write operation calls, thus Active
> NameNode enter a state of false death for number of seconds even for minutes.
> The similar performance concerns of Balancer have reported by HDFS-9412,
> HDFS-7967, etc.
> If Standby NameNode can shoulder #getBlocks heavy burden, it could speed up
> the progress of balancing and reduce performance impact to Active NameNode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13114) CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
[
https://issues.apache.org/jira/browse/HDFS-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-13114:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
3.1.0
Status: Resolved (was: Patch Available)
Thanks [~hanishakoneru] for the contribution and all for the reviews. I've
commited the patch to trunk, branch-3.1 and branch-3.2
> CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
>
>
> Key: HDFS-13114
> URL: https://issues.apache.org/jira/browse/HDFS-13114
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Fix For: 3.1.0, 3.2.0
>
> Attachments: HDFS-13114.001.patch
>
>
> The {{crypto -reencryptZone -path }} command takes in a path
> argument. But when creating {{HdfsAdmin}} object, it takes the defaultFs
> instead of resolving from the path. This causes the following exception if
> the authority component in path does not match the authority of default Fs.
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://mycluster-node-1:8020/zone1
> IllegalArgumentException: Wrong FS: hdfs://mycluster-node-1:8020/zone1,
> expected: hdfs://ns1{code}
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://ns2/zone2
> IllegalArgumentException: Wrong FS: hdfs://ns2/zone2, expected:
> hdfs://ns1{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13114) CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
[
https://issues.apache.org/jira/browse/HDFS-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-13114:
--
Fix Version/s: 3.0.2
> CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
>
>
> Key: HDFS-13114
> URL: https://issues.apache.org/jira/browse/HDFS-13114
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Fix For: 3.1.0, 3.0.2, 3.2.0
>
> Attachments: HDFS-13114.001.patch
>
>
> The {{crypto -reencryptZone -path }} command takes in a path
> argument. But when creating {{HdfsAdmin}} object, it takes the defaultFs
> instead of resolving from the path. This causes the following exception if
> the authority component in path does not match the authority of default Fs.
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://mycluster-node-1:8020/zone1
> IllegalArgumentException: Wrong FS: hdfs://mycluster-node-1:8020/zone1,
> expected: hdfs://ns1{code}
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://ns2/zone2
> IllegalArgumentException: Wrong FS: hdfs://ns2/zone2, expected:
> hdfs://ns1{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13114) CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
[
https://issues.apache.org/jira/browse/HDFS-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380637#comment-16380637
]
Xiaoyu Yao edited comment on HDFS-13114 at 2/28/18 4:47 PM:
Thanks [~hanishakoneru] for the contribution and all for the reviews. I've
commited the patch to trunk, branch-3.1 and branch-3.0
was (Author: xyao):
Thanks [~hanishakoneru] for the contribution and all for the reviews. I've
commited the patch to trunk, branch-3.1 and branch-3.2
> CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
>
>
> Key: HDFS-13114
> URL: https://issues.apache.org/jira/browse/HDFS-13114
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Fix For: 3.1.0, 3.0.2, 3.2.0
>
> Attachments: HDFS-13114.001.patch
>
>
> The {{crypto -reencryptZone -path }} command takes in a path
> argument. But when creating {{HdfsAdmin}} object, it takes the defaultFs
> instead of resolving from the path. This causes the following exception if
> the authority component in path does not match the authority of default Fs.
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://mycluster-node-1:8020/zone1
> IllegalArgumentException: Wrong FS: hdfs://mycluster-node-1:8020/zone1,
> expected: hdfs://ns1{code}
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://ns2/zone2
> IllegalArgumentException: Wrong FS: hdfs://ns2/zone2, expected:
> hdfs://ns1{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13114) CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
[
https://issues.apache.org/jira/browse/HDFS-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380662#comment-16380662
]
Hudson commented on HDFS-13114:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13737 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13737/])
HDFS-13114. CryptoAdmin#ReencryptZoneCommand should resolve Namespace (xyao:
rev 2574375bf571b1fbc15ede7ba3b2826dc1dd3e70)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/CryptoAdmin.java
> CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
>
>
> Key: HDFS-13114
> URL: https://issues.apache.org/jira/browse/HDFS-13114
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Fix For: 3.1.0, 3.0.2, 3.2.0
>
> Attachments: HDFS-13114.001.patch
>
>
> The {{crypto -reencryptZone -path }} command takes in a path
> argument. But when creating {{HdfsAdmin}} object, it takes the defaultFs
> instead of resolving from the path. This causes the following exception if
> the authority component in path does not match the authority of default Fs.
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://mycluster-node-1:8020/zone1
> IllegalArgumentException: Wrong FS: hdfs://mycluster-node-1:8020/zone1,
> expected: hdfs://ns1{code}
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://ns2/zone2
> IllegalArgumentException: Wrong FS: hdfs://ns2/zone2, expected:
> hdfs://ns1{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380685#comment-16380685 ] Íñigo Goiri commented on HDFS-13204: Thanks [~liuhongtong] for opening this. I think it makes sense to switch the icon; I'm used to the wrench icon but I guess it doesn't make much sense. I never gave too much thought to the UI itself; we may want to take a full pass on the icons, etc. For example, the safe mode for the Router is more like the Standby. What others think? [~ywskycn], [~linyiqun]? > RBF: optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13204: --- Summary: RBF: Optimize name service safe mode icon (was: RBF: optimize name service safe mode icon) > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12284) RBF: Support for Kerberos authentication and delegation tokens
[ https://issues.apache.org/jira/browse/HDFS-12284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12284: --- Summary: RBF: Support for Kerberos authentication and delegation tokens (was: Router support for Kerberos authentication and delegation tokens) > RBF: Support for Kerberos authentication and delegation tokens > -- > > Key: HDFS-12284 > URL: https://issues.apache.org/jira/browse/HDFS-12284 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: security >Reporter: Zhe Zhang >Assignee: Sherwood Zheng >Priority: Major > Fix For: HDFS-10467 > > > HDFS Router should support Kerberos authentication and issuing / managing > HDFS delegation tokens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13081) Datanode#checkSecureConfig should check HTTPS and SASL encryption
[ https://issues.apache.org/jira/browse/HDFS-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380709#comment-16380709 ] Xiaoyu Yao commented on HDFS-13081: --- Thanks [~ajayydv] for the update. +1 for the latest patch. There is a checkstyle warning on unused import that I can remove at commit. > Datanode#checkSecureConfig should check HTTPS and SASL encryption > - > > Key: HDFS-13081 > URL: https://issues.apache.org/jira/browse/HDFS-13081 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, security >Affects Versions: 3.0.0 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13081.000.patch, HDFS-13081.001.patch, > HDFS-13081.002.patch, HDFS-13081.003.patch, HDFS-13081.004.patch, > HDFS-13081.005.patch, HDFS-13081.006.patch > > > Datanode#checkSecureConfig currently check the following to determine if > secure datanode is enabled. > # The server has bound to privileged ports for RPC and HTTP via > SecureDataNodeStarter. > # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain > HTTP) for the HTTP server. The SASL handshake guarantees authentication of > the RPC server before a client transmits a secret, such as a block access > token. Similarly, SSL guarantees authentication of the > HTTP server before a client transmits a secret, such as a delegation token. > For the 2nd case, HTTPS_ONLY means all the traffic between REST client/server > will be encrypted. However, the logic to check only if SASL property resolver > is configured does not mean server requires an encrypted RPC. > This ticket is open to further check and ensure datanode SASL property > resolver has a QoP that includes auth-conf(PRIVACY). Note that the SASL QoP > (Quality of Protection) negotiation may drop RPC protection level from > auth-conf(PRIVACY) to auth-int(integrity) or auth(authentication) only, which > should be fine by design. > > cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13081) Datanode#checkSecureConfig should allow privileged HTTP and SASL encryption
[ https://issues.apache.org/jira/browse/HDFS-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13081: -- Summary: Datanode#checkSecureConfig should allow privileged HTTP and SASL encryption (was: Datanode#checkSecureConfig should check HTTPS and SASL encryption) > Datanode#checkSecureConfig should allow privileged HTTP and SASL encryption > --- > > Key: HDFS-13081 > URL: https://issues.apache.org/jira/browse/HDFS-13081 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, security >Affects Versions: 3.0.0 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13081.000.patch, HDFS-13081.001.patch, > HDFS-13081.002.patch, HDFS-13081.003.patch, HDFS-13081.004.patch, > HDFS-13081.005.patch, HDFS-13081.006.patch > > > Datanode#checkSecureConfig currently check the following to determine if > secure datanode is enabled. > # The server has bound to privileged ports for RPC and HTTP via > SecureDataNodeStarter. > # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain > HTTP) for the HTTP server. The SASL handshake guarantees authentication of > the RPC server before a client transmits a secret, such as a block access > token. Similarly, SSL guarantees authentication of the > HTTP server before a client transmits a secret, such as a delegation token. > For the 2nd case, HTTPS_ONLY means all the traffic between REST client/server > will be encrypted. However, the logic to check only if SASL property resolver > is configured does not mean server requires an encrypted RPC. > This ticket is open to further check and ensure datanode SASL property > resolver has a QoP that includes auth-conf(PRIVACY). Note that the SASL QoP > (Quality of Protection) negotiation may drop RPC protection level from > auth-conf(PRIVACY) to auth-int(integrity) or auth(authentication) only, which > should be fine by design. > > cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13081) Datanode#checkSecureConfig should allow privileged HTTP and SASL encryption
[ https://issues.apache.org/jira/browse/HDFS-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13081: -- Description: Datanode#checkSecureConfig currently check the following to determine if secure datanode is enabled. # The server has bound to privileged ports for RPC and HTTP via SecureDataNodeStarter. # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain HTTP) for the HTTP server. Auththentication of RPC server The SASL handshake guarantees authentication of the RPC server before a client transmits a secret, such as a block access token. Similar can be achieved with a privileged RPC port. Similarly, SSL guarantees authentication of the HTTP server before a client transmits a secret, such as a delegation token. This ticket is open to allow privileged HTTP as an alternative to HTTPS to work with SASL. cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. was: Datanode#checkSecureConfig currently check the following to determine if secure datanode is enabled. # The server has bound to privileged ports for RPC and HTTP via SecureDataNodeStarter. # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain HTTP) for the HTTP server. The SASL handshake guarantees authentication of the RPC server before a client transmits a secret, such as a block access token. Similarly, SSL guarantees authentication of the HTTP server before a client transmits a secret, such as a delegation token. For the 2nd case, HTTPS_ONLY means all the traffic between REST client/server will be encrypted. However, the logic to check only if SASL property resolver is configured does not mean server requires an encrypted RPC. This ticket is open to further check and ensure datanode SASL property resolver has a QoP that includes auth-conf(PRIVACY). Note that the SASL QoP (Quality of Protection) negotiation may drop RPC protection level from auth-conf(PRIVACY) to auth-int(integrity) or auth(authentication) only, which should be fine by design. cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > Datanode#checkSecureConfig should allow privileged HTTP and SASL encryption > --- > > Key: HDFS-13081 > URL: https://issues.apache.org/jira/browse/HDFS-13081 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, security >Affects Versions: 3.0.0 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13081.000.patch, HDFS-13081.001.patch, > HDFS-13081.002.patch, HDFS-13081.003.patch, HDFS-13081.004.patch, > HDFS-13081.005.patch, HDFS-13081.006.patch > > > Datanode#checkSecureConfig currently check the following to determine if > secure datanode is enabled. > # The server has bound to privileged ports for RPC and HTTP via > SecureDataNodeStarter. > # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain > HTTP) for the HTTP server. > Auththentication of RPC server > The SASL handshake guarantees authentication of the RPC server before a > client transmits a secret, such as a block access token. Similar can be > achieved with a privileged RPC port. > Similarly, SSL guarantees authentication of the HTTP server before a client > transmits a secret, such as a delegation token. > This ticket is open to allow privileged HTTP as an alternative to HTTPS to > work with SASL. > > cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13081) Datanode#checkSecureConfig should allow privileged HTTP and SASL encryption
[ https://issues.apache.org/jira/browse/HDFS-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13081: -- Description: Datanode#checkSecureConfig currently check the following to determine if secure datanode is enabled. # The server has bound to privileged ports for RPC and HTTP via SecureDataNodeStarter. # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain HTTP) for the HTTP server. Authentication of Datanode RPC server can be done either via SASL handshake or JSVC/privilege RPC port. This guarantees authentication of the datanode RPC server before a client transmits a secret, such as a block access token. Authentication of the HTTP server can also be done either via HTTPS/SSL or JSVC/privilege HTTP port. This guarantees authentication of datandoe HTTP server before a client transmits a secret, such as a delegation token. This ticket is open to allow privileged HTTP as an alternative to HTTPS to work with SASL based RPC protection. cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. was: Datanode#checkSecureConfig currently check the following to determine if secure datanode is enabled. # The server has bound to privileged ports for RPC and HTTP via SecureDataNodeStarter. # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain HTTP) for the HTTP server. Auththentication of RPC server The SASL handshake guarantees authentication of the RPC server before a client transmits a secret, such as a block access token. Similar can be achieved with a privileged RPC port. Similarly, SSL guarantees authentication of the HTTP server before a client transmits a secret, such as a delegation token. This ticket is open to allow privileged HTTP as an alternative to HTTPS to work with SASL. cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > Datanode#checkSecureConfig should allow privileged HTTP and SASL encryption > --- > > Key: HDFS-13081 > URL: https://issues.apache.org/jira/browse/HDFS-13081 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, security >Affects Versions: 3.0.0 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13081.000.patch, HDFS-13081.001.patch, > HDFS-13081.002.patch, HDFS-13081.003.patch, HDFS-13081.004.patch, > HDFS-13081.005.patch, HDFS-13081.006.patch > > > Datanode#checkSecureConfig currently check the following to determine if > secure datanode is enabled. > # The server has bound to privileged ports for RPC and HTTP via > SecureDataNodeStarter. > # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain > HTTP) for the HTTP server. > Authentication of Datanode RPC server can be done either via SASL handshake > or JSVC/privilege RPC port. > This guarantees authentication of the datanode RPC server before a client > transmits a secret, such as a block access token. > Authentication of the HTTP server can also be done either via HTTPS/SSL or > JSVC/privilege HTTP port. This guarantees authentication of datandoe HTTP > server before a client transmits a secret, such as a delegation token. > This ticket is open to allow privileged HTTP as an alternative to HTTPS to > work with SASL based RPC protection. > > cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13081) Datanode#checkSecureConfig should allow SASL and privileged HTTP
[ https://issues.apache.org/jira/browse/HDFS-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13081: -- Summary: Datanode#checkSecureConfig should allow SASL and privileged HTTP (was: Datanode#checkSecureConfig should allow privileged HTTP and SASL encryption) > Datanode#checkSecureConfig should allow SASL and privileged HTTP > > > Key: HDFS-13081 > URL: https://issues.apache.org/jira/browse/HDFS-13081 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, security >Affects Versions: 3.0.0 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13081.000.patch, HDFS-13081.001.patch, > HDFS-13081.002.patch, HDFS-13081.003.patch, HDFS-13081.004.patch, > HDFS-13081.005.patch, HDFS-13081.006.patch > > > Datanode#checkSecureConfig currently check the following to determine if > secure datanode is enabled. > # The server has bound to privileged ports for RPC and HTTP via > SecureDataNodeStarter. > # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain > HTTP) for the HTTP server. > Authentication of Datanode RPC server can be done either via SASL handshake > or JSVC/privilege RPC port. > This guarantees authentication of the datanode RPC server before a client > transmits a secret, such as a block access token. > Authentication of the HTTP server can also be done either via HTTPS/SSL or > JSVC/privilege HTTP port. This guarantees authentication of datandoe HTTP > server before a client transmits a secret, such as a delegation token. > This ticket is open to allow privileged HTTP as an alternative to HTTPS to > work with SASL based RPC protection. > > cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13081) Datanode#checkSecureConfig should allow SASL and privileged HTTP
[ https://issues.apache.org/jira/browse/HDFS-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380736#comment-16380736 ] Xiaoyu Yao commented on HDFS-13081: --- Update the title and description, I will commit the patch shortly. > Datanode#checkSecureConfig should allow SASL and privileged HTTP > > > Key: HDFS-13081 > URL: https://issues.apache.org/jira/browse/HDFS-13081 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, security >Affects Versions: 3.0.0 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Attachments: HDFS-13081.000.patch, HDFS-13081.001.patch, > HDFS-13081.002.patch, HDFS-13081.003.patch, HDFS-13081.004.patch, > HDFS-13081.005.patch, HDFS-13081.006.patch > > > Datanode#checkSecureConfig currently check the following to determine if > secure datanode is enabled. > # The server has bound to privileged ports for RPC and HTTP via > SecureDataNodeStarter. > # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain > HTTP) for the HTTP server. > Authentication of Datanode RPC server can be done either via SASL handshake > or JSVC/privilege RPC port. > This guarantees authentication of the datanode RPC server before a client > transmits a secret, such as a block access token. > Authentication of the HTTP server can also be done either via HTTPS/SSL or > JSVC/privilege HTTP port. This guarantees authentication of datandoe HTTP > server before a client transmits a secret, such as a delegation token. > This ticket is open to allow privileged HTTP as an alternative to HTTPS to > work with SASL based RPC protection. > > cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[
https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380794#comment-16380794
]
Tsz Wo Nicholas Sze commented on HDFS-13102:
The 008 patch looks good.
- DirectoryDiffList.getLevel is not used. Please remove it.
- DirectoryDiffList.skipNodeList should be final.
- There are a few checkstyle warnings. Please fix them.
-* For the checkstyle warning on iterator(), it could be fixed by moving i as a
local variable, i.e.
{code}
@Override
public Iterator iterator() {
final Iterator i = skipNodeList.iterator();
return new Iterator() {
@Override public boolean hasNext() {
return i.hasNext();
}
@Override public DirectoryDiff next() {
return i.next().getDiff();
}
};
}
{code}
> Implement SnapshotSkipList class to store Multi level DirectoryDiffs
>
>
> Key: HDFS-13102
> URL: https://issues.apache.org/jira/browse/HDFS-13102
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-13102.001.patch, HDFS-13102.002.patch,
> HDFS-13102.003.patch, HDFS-13102.004.patch, HDFS-13102.005.patch,
> HDFS-13102.006.patch, HDFS-13102.007.patch, HDFS-13102.008.patch
>
>
> HDFS-11225 explains an issue where deletion of older snapshots can take a
> very long time in case the no of snapshot diffs is quite large for
> directories. For any directory under a snapshot, to construct the children
> list , it needs to combine all the diffs from that particular snapshot to the
> last snapshotDiff record and reverseApply to the current children list of the
> directory on live fs. This can take a significant time if the no of snapshot
> diffs are quite large and changes per diff is significant.
> This Jira proposes to store the Directory diffs in a SnapshotSkip list, where
> we store multi level DirectoryDiffs. At each level, the Directory Diff will
> be cumulative diff of k snapshot diffs,
> where k is the level of a node in the list.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13081) Datanode#checkSecureConfig should allow SASL and privileged HTTP
[ https://issues.apache.org/jira/browse/HDFS-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-13081: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 3.0.2 3.1.0 Status: Resolved (was: Patch Available) Thanks [~ajayydv] for the contribution and all for the discussions and reviews. I've committed the patch to trunk, branch-3.1 and branch-3.0. > Datanode#checkSecureConfig should allow SASL and privileged HTTP > > > Key: HDFS-13081 > URL: https://issues.apache.org/jira/browse/HDFS-13081 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, security >Affects Versions: 3.0.0 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.1.0, 3.0.2, 3.2.0 > > Attachments: HDFS-13081.000.patch, HDFS-13081.001.patch, > HDFS-13081.002.patch, HDFS-13081.003.patch, HDFS-13081.004.patch, > HDFS-13081.005.patch, HDFS-13081.006.patch > > > Datanode#checkSecureConfig currently check the following to determine if > secure datanode is enabled. > # The server has bound to privileged ports for RPC and HTTP via > SecureDataNodeStarter. > # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain > HTTP) for the HTTP server. > Authentication of Datanode RPC server can be done either via SASL handshake > or JSVC/privilege RPC port. > This guarantees authentication of the datanode RPC server before a client > transmits a secret, such as a block access token. > Authentication of the HTTP server can also be done either via HTTPS/SSL or > JSVC/privilege HTTP port. This guarantees authentication of datandoe HTTP > server before a client transmits a secret, such as a delegation token. > This ticket is open to allow privileged HTTP as an alternative to HTTPS to > work with SASL based RPC protection. > > cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13206) IllegalStateException: Unable to finalize edits file
Ted Yu created HDFS-13206:
-
Summary: IllegalStateException: Unable to finalize edits file
Key: HDFS-13206
URL: https://issues.apache.org/jira/browse/HDFS-13206
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ted Yu
I noticed the following in hbase test output running against hadoop3:
{code}
2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402):
Error: finalize log segment 1, 658 failed for (journal
JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
stream=null))
java.lang.IllegalStateException: Unable to finalize edits file
/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
at
org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
at
org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
at
org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
at
org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
at
org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
at
org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
at
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
at
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
at
org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13206) IllegalStateException: Unable to finalize edits file
[
https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HDFS-13206:
--
Attachment: testFavoredNodeTableImport-output.txt
> IllegalStateException: Unable to finalize edits file
>
>
> Key: HDFS-13206
> URL: https://issues.apache.org/jira/browse/HDFS-13206
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: testFavoredNodeTableImport-output.txt
>
>
> I noticed the following in hbase test output running against hadoop3:
> {code}
> 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402):
> Error: finalize log segment 1, 658 failed for (journal
> JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
> stream=null))
> java.lang.IllegalStateException: Unable to finalize edits file
> /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
> at
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
> at
> org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[ https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shashikant Banerjee updated HDFS-13102: --- Attachment: HDFS-13102.009.patch > Implement SnapshotSkipList class to store Multi level DirectoryDiffs > > > Key: HDFS-13102 > URL: https://issues.apache.org/jira/browse/HDFS-13102 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13102.001.patch, HDFS-13102.002.patch, > HDFS-13102.003.patch, HDFS-13102.004.patch, HDFS-13102.005.patch, > HDFS-13102.006.patch, HDFS-13102.007.patch, HDFS-13102.008.patch, > HDFS-13102.009.patch > > > HDFS-11225 explains an issue where deletion of older snapshots can take a > very long time in case the no of snapshot diffs is quite large for > directories. For any directory under a snapshot, to construct the children > list , it needs to combine all the diffs from that particular snapshot to the > last snapshotDiff record and reverseApply to the current children list of the > directory on live fs. This can take a significant time if the no of snapshot > diffs are quite large and changes per diff is significant. > This Jira proposes to store the Directory diffs in a SnapshotSkip list, where > we store multi level DirectoryDiffs. At each level, the Directory Diff will > be cumulative diff of k snapshot diffs, > where k is the level of a node in the list. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[ https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380814#comment-16380814 ] Shashikant Banerjee commented on HDFS-13102: Thanks [~szetszwo], for the review . Patch v9 addresses your review comments. Please have a look. > Implement SnapshotSkipList class to store Multi level DirectoryDiffs > > > Key: HDFS-13102 > URL: https://issues.apache.org/jira/browse/HDFS-13102 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13102.001.patch, HDFS-13102.002.patch, > HDFS-13102.003.patch, HDFS-13102.004.patch, HDFS-13102.005.patch, > HDFS-13102.006.patch, HDFS-13102.007.patch, HDFS-13102.008.patch, > HDFS-13102.009.patch > > > HDFS-11225 explains an issue where deletion of older snapshots can take a > very long time in case the no of snapshot diffs is quite large for > directories. For any directory under a snapshot, to construct the children > list , it needs to combine all the diffs from that particular snapshot to the > last snapshotDiff record and reverseApply to the current children list of the > directory on live fs. This can take a significant time if the no of snapshot > diffs are quite large and changes per diff is significant. > This Jira proposes to store the Directory diffs in a SnapshotSkip list, where > we store multi level DirectoryDiffs. At each level, the Directory Diff will > be cumulative diff of k snapshot diffs, > where k is the level of a node in the list. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13081) Datanode#checkSecureConfig should allow SASL and privileged HTTP
[ https://issues.apache.org/jira/browse/HDFS-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380819#comment-16380819 ] Hudson commented on HDFS-13081: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13738 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13738/]) HDFS-13081. Datanode#checkSecureConfig should allow SASL and privileged (xyao: rev f20e10b2dd59f99d9af00960a067b9893e69) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/SecureMode.md * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/TestSaslDataTransfer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/SecureDataNodeStarter.java > Datanode#checkSecureConfig should allow SASL and privileged HTTP > > > Key: HDFS-13081 > URL: https://issues.apache.org/jira/browse/HDFS-13081 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, security >Affects Versions: 3.0.0 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.1.0, 3.0.2, 3.2.0 > > Attachments: HDFS-13081.000.patch, HDFS-13081.001.patch, > HDFS-13081.002.patch, HDFS-13081.003.patch, HDFS-13081.004.patch, > HDFS-13081.005.patch, HDFS-13081.006.patch > > > Datanode#checkSecureConfig currently check the following to determine if > secure datanode is enabled. > # The server has bound to privileged ports for RPC and HTTP via > SecureDataNodeStarter. > # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain > HTTP) for the HTTP server. > Authentication of Datanode RPC server can be done either via SASL handshake > or JSVC/privilege RPC port. > This guarantees authentication of the datanode RPC server before a client > transmits a secret, such as a block access token. > Authentication of the HTTP server can also be done either via HTTPS/SSL or > JSVC/privilege HTTP port. This guarantees authentication of datandoe HTTP > server before a client transmits a secret, such as a delegation token. > This ticket is open to allow privileged HTTP as an alternative to HTTPS to > work with SASL based RPC protection. > > cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13206) IllegalStateException: Unable to finalize edits file
[
https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380822#comment-16380822
]
Kihwal Lee commented on HDFS-13206:
---
It looks like the NN went into safe mode because the disk ran out of space.
> IllegalStateException: Unable to finalize edits file
>
>
> Key: HDFS-13206
> URL: https://issues.apache.org/jira/browse/HDFS-13206
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: testFavoredNodeTableImport-output.txt
>
>
> I noticed the following in hbase test output running against hadoop3:
> {code}
> 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402):
> Error: finalize log segment 1, 658 failed for (journal
> JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
> stream=null))
> java.lang.IllegalStateException: Unable to finalize edits file
> /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
> at
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
> at
> org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13206) IllegalStateException: Unable to finalize edits file
[
https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380822#comment-16380822
]
Kihwal Lee edited comment on HDFS-13206 at 2/28/18 6:52 PM:
It looks like the NN went into safe mode because the disk ran out of space.
Check the test work space.
was (Author: kihwal):
It looks like the NN went into safe mode because the disk ran out of space.
> IllegalStateException: Unable to finalize edits file
>
>
> Key: HDFS-13206
> URL: https://issues.apache.org/jira/browse/HDFS-13206
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: testFavoredNodeTableImport-output.txt
>
>
> I noticed the following in hbase test output running against hadoop3:
> {code}
> 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402):
> Error: finalize log segment 1, 658 failed for (journal
> JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
> stream=null))
> java.lang.IllegalStateException: Unable to finalize edits file
> /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
> at
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
> at
> org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13206) IllegalStateException: Unable to finalize edits file
[
https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380829#comment-16380829
]
Ted Yu commented on HDFS-13206:
---
I don't think that was the case.
I used {{df}} command on the Linux where the test was run.
No partition was nearly half used.
> IllegalStateException: Unable to finalize edits file
>
>
> Key: HDFS-13206
> URL: https://issues.apache.org/jira/browse/HDFS-13206
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: testFavoredNodeTableImport-output.txt
>
>
> I noticed the following in hbase test output running against hadoop3:
> {code}
> 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402):
> Error: finalize log segment 1, 658 failed for (journal
> JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
> stream=null))
> java.lang.IllegalStateException: Unable to finalize edits file
> /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
> at
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
> at
> org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13166) [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls
[
https://issues.apache.org/jira/browse/HDFS-13166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380845#comment-16380845
]
Surendra Singh Lilhore commented on HDFS-13166:
---
+1 LGTM
> [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly
> getLiveDatanodeStorageReport() calls
> -
>
> Key: HDFS-13166
> URL: https://issues.apache.org/jira/browse/HDFS-13166
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Attachments: HDFS-13166-HDFS-10285-00.patch,
> HDFS-13166-HDFS-10285-01.patch, HDFS-13166-HDFS-10285-02.patch,
> HDFS-13166-HDFS-10285-03.patch
>
>
> Presently {{#getLiveDatanodeStorageReport()}} is fetched for every file and
> does the computation. This Jira sub-task is to discuss and implement a cache
> mechanism which in turn reduces the number of function calls. Also, could
> define a configurable refresh interval and periodically refresh the DN cache
> by fetching latest {{#getLiveDatanodeStorageReport}} on this interval.
> Following comments taken from HDFS-10285,
> [here|https://issues.apache.org/jira/browse/HDFS-10285?focusedCommentId=16347472&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16347472]
> Comment-7)
> {quote}Adding getDatanodeStorageReport is concerning.
> getDatanodeListForReport is already a very bad method that should be avoided
> for anything but jmx – even then it’s a concern. I eliminated calls to it
> years ago. All it takes is a nscd/dns hiccup and you’re left holding the fsn
> lock for an excessive length of time. Beyond that, the response is going to
> be pretty large and tagging all the storage reports is not going to be cheap.
> verifyTargetDatanodeHasSpaceForScheduling does it really need the namesystem
> lock? Can’t DatanodeDescriptor#chooseStorage4Block synchronize on its
> storageMap?
> Appears to be calling getLiveDatanodeStorageReport for every file. As
> mentioned earlier, this is NOT cheap. The SPS should be able to operate on a
> fuzzy/cached state of the world. Then it gets another datanode report to
> determine the number of live nodes to decide if it should sleep before
> processing the next path. The number of nodes from the prior cached view of
> the world should suffice.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380851#comment-16380851 ] Wei Yan commented on HDFS-13204: The "Safe Mode" icon under tab "Subclusters" --> "Nameservice Information" may also need to be changed. But I have little confused here, does nameservice also has the 4 states (Active,Standby, Safe mode, Unavailable) like NameNode. One more icon needs to change is the "Decommissioned" icon under tab "Datanodes", which can be the same as current what we use in the NameNode WebUI. > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13166) [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly getLiveDatanodeStorageReport() calls
[
https://issues.apache.org/jira/browse/HDFS-13166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-13166:
--
Resolution: Fixed
Fix Version/s: HDFS-10285
Status: Resolved (was: Patch Available)
Committed to branch. Thanks [~rakeshr]
> [SPS]: Implement caching mechanism to keep LIVE datanodes to minimize costly
> getLiveDatanodeStorageReport() calls
> -
>
> Key: HDFS-13166
> URL: https://issues.apache.org/jira/browse/HDFS-13166
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Rakesh R
>Assignee: Rakesh R
>Priority: Major
> Fix For: HDFS-10285
>
> Attachments: HDFS-13166-HDFS-10285-00.patch,
> HDFS-13166-HDFS-10285-01.patch, HDFS-13166-HDFS-10285-02.patch,
> HDFS-13166-HDFS-10285-03.patch
>
>
> Presently {{#getLiveDatanodeStorageReport()}} is fetched for every file and
> does the computation. This Jira sub-task is to discuss and implement a cache
> mechanism which in turn reduces the number of function calls. Also, could
> define a configurable refresh interval and periodically refresh the DN cache
> by fetching latest {{#getLiveDatanodeStorageReport}} on this interval.
> Following comments taken from HDFS-10285,
> [here|https://issues.apache.org/jira/browse/HDFS-10285?focusedCommentId=16347472&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16347472]
> Comment-7)
> {quote}Adding getDatanodeStorageReport is concerning.
> getDatanodeListForReport is already a very bad method that should be avoided
> for anything but jmx – even then it’s a concern. I eliminated calls to it
> years ago. All it takes is a nscd/dns hiccup and you’re left holding the fsn
> lock for an excessive length of time. Beyond that, the response is going to
> be pretty large and tagging all the storage reports is not going to be cheap.
> verifyTargetDatanodeHasSpaceForScheduling does it really need the namesystem
> lock? Can’t DatanodeDescriptor#chooseStorage4Block synchronize on its
> storageMap?
> Appears to be calling getLiveDatanodeStorageReport for every file. As
> mentioned earlier, this is NOT cheap. The SPS should be able to operate on a
> fuzzy/cached state of the world. Then it gets another datanode report to
> determine the number of live nodes to decide if it should sleep before
> processing the next path. The number of nodes from the prior cached view of
> the world should suffice.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13171) Handle Deletion of nodes in SnasphotSkipList
[
https://issues.apache.org/jira/browse/HDFS-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380859#comment-16380859
]
Tsz Wo Nicholas Sze commented on HDFS-13171:
How to apply the patch? It somehow requires the HDFS-13102.008.patch file.
{code}
++ git apply --whitespace=fix HDFS-13171.000.patch
HDFS-13171.000.patch:137: trailing whitespace.
+
error: ../../patches/HDFS-13102.008.patch: No such file or directory
{code}
The usual way to generate the patch is to first commit HDFS-13102.008.patch
locally.
> Handle Deletion of nodes in SnasphotSkipList
>
>
> Key: HDFS-13171
> URL: https://issues.apache.org/jira/browse/HDFS-13171
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-13171.000.patch
>
>
> This Jira will handle deletion of skipListNodes from DirectoryDiffList . If a
> node has multiple levels, the list needs to be balanced .If the node is uni
> level, no balancing of the list is required.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13206) IllegalStateException: Unable to finalize edits file
[
https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380888#comment-16380888
]
Rushabh S Shah commented on HDFS-13206:
---
{noformat}
2018-02-28 18:39:59,969 WARN
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@1a7d7a7e]
namenode.NameNodeResourceChecker$CheckedVolume(89): Space available on volume
'/dev/sdc1' is 0, which is below the configured reserved amount 104857600
2018-02-28 18:39:59,970 WARN
[org.apache.hadoop.hdfs.server.namenode.FSNamesystem$NameNodeResourceMonitor@1a7d7a7e]
namenode.FSNamesystem$NameNodeResourceMonitor(3847): NameNode low on available
disk space. Entering safe mode.
{noformat}
Check {{/dev/sdc1}} volume.
> IllegalStateException: Unable to finalize edits file
>
>
> Key: HDFS-13206
> URL: https://issues.apache.org/jira/browse/HDFS-13206
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: testFavoredNodeTableImport-output.txt
>
>
> I noticed the following in hbase test output running against hadoop3:
> {code}
> 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402):
> Error: finalize log segment 1, 658 failed for (journal
> JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
> stream=null))
> java.lang.IllegalStateException: Unable to finalize edits file
> /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
> at
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
> at
> org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13206) IllegalStateException: Unable to finalize edits file
[
https://issues.apache.org/jira/browse/HDFS-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380902#comment-16380902
]
Ted Yu commented on HDFS-13206:
---
{code}
devtmpfs 65906460 0 65906460 0% /dev
tmpfs 65914268 0 65914268 0% /dev/shm
{code}
> IllegalStateException: Unable to finalize edits file
>
>
> Key: HDFS-13206
> URL: https://issues.apache.org/jira/browse/HDFS-13206
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Priority: Minor
> Attachments: testFavoredNodeTableImport-output.txt
>
>
> I noticed the following in hbase test output running against hadoop3:
> {code}
> 2018-02-28 18:40:18,491 ERROR [Time-limited test] namenode.JournalSet(402):
> Error: finalize log segment 1, 658 failed for (journal
> JournalAndStream(mgr=FileJournalManager(root=/mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1),
> stream=null))
> java.lang.IllegalStateException: Unable to finalize edits file
> /mnt/disk2/a/2-hbase/hbase-server/target/test-data/5670112c-31f1-43b0-af31-c1182e142e63/cluster_8f993609-c3a1-4fb4-8b3d-0e642261deb1/dfs/name-0-1/current/edits_inprogress_001
> at
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.finalizeLogSegment(FileJournalManager.java:153)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet$2.apply(JournalSet.java:224)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:385)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.finalizeLogSegment(JournalSet.java:219)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.endCurrentLogSegment(FSEditLog.java:1427)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.close(FSEditLog.java:398)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogAsync.close(FSEditLogAsync.java:110)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.stopActiveServices(FSNamesystem.java:1320)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.stopActiveServices(NameNode.java:1909)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.exitState(ActiveState.java:70)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:1013)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.stopAndJoinNameNode(MiniDFSCluster.java:2047)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1987)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1958)
> at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1951)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniDFSCluster(HBaseTestingUtility.java:767)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:1109)
> at
> org.apache.hadoop.hbase.master.balancer.TestFavoredNodeTableImport.stopCluster(TestFavoredNodeTableImport.java:71)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[ https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-13207 started by Bharat Viswanadham. - > Add blockpool used for DFSAdmin Report command > --- > > Key: HDFS-13207 > URL: https://issues.apache.org/jira/browse/HDFS-13207 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > Add block pool used for dfsadmin -report command. > This will be helpful on a federated cluster to know DFS used by a particular > namespace. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13207) Add blockpool used for DFSAdmin Report command
Bharat Viswanadham created HDFS-13207: - Summary: Add blockpool used for DFSAdmin Report command Key: HDFS-13207 URL: https://issues.apache.org/jira/browse/HDFS-13207 Project: Hadoop HDFS Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham Add block pool used for dfsadmin -report command. This will be helpful on a federated cluster to know DFS used by a particular namespace. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[ https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13207: -- Description: Add block pool used for dfsadmin -report command. This will be helpful on a federated cluster to know DFS used by a particular namespace. On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same. was: Add block pool used for dfsadmin -report command. This will be helpful on a federated cluster to know DFS used by a particular namespace. > Add blockpool used for DFSAdmin Report command > --- > > Key: HDFS-13207 > URL: https://issues.apache.org/jira/browse/HDFS-13207 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > Add block pool used for dfsadmin -report command. > This will be helpful on a federated cluster to know DFS used by a particular > namespace. > > On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[ https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13207: -- Issue Type: Improvement (was: Bug) > Add blockpool used for DFSAdmin Report command > --- > > Key: HDFS-13207 > URL: https://issues.apache.org/jira/browse/HDFS-13207 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > Add block pool used for dfsadmin -report command. > This will be helpful on a federated cluster to know DFS used by a particular > namespace. > > On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[ https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13207: -- Attachment: HDFS-13207.00.patch > Add blockpool used for DFSAdmin Report command > --- > > Key: HDFS-13207 > URL: https://issues.apache.org/jira/browse/HDFS-13207 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13207.00.patch > > > Add block pool used for dfsadmin -report command. > This will be helpful on a federated cluster to know DFS used by a particular > namespace. > > On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[ https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13207: -- Status: Patch Available (was: In Progress) > Add blockpool used for DFSAdmin Report command > --- > > Key: HDFS-13207 > URL: https://issues.apache.org/jira/browse/HDFS-13207 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13207.00.patch > > > Add block pool used for dfsadmin -report command. > This will be helpful on a federated cluster to know DFS used by a particular > namespace. > > On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[
https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16380976#comment-16380976
]
Bharat Viswanadham commented on HDFS-13207:
---
Existing TestDFSAdmin test cases, covers this.
Tested it on a Federated cluster.
The output is as below:
{code:java}
hdfs@373dc353753f:/$ /hadoop/bin/hdfs dfsadmin -report
2018-02-28 20:15:14,153 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Configured Capacity: 1995884724224 (1.82 TB)
Present Capacity: 1117203682022 (1.02 TB)
DFS Remaining: 1117203578880 (1.02 TB)
DFS Used: 103142 (100.72 KB)
DFS Used%: 0.00%
Replicated Blocks:
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Pending deletion blocks: 0
Erasure Coded Block Groups:
Low redundancy block groups: 0
Block groups with corrupt internal blocks: 0
Missing block groups: 0
Pending deletion blocks: 0
-
Live datanodes (4):
Name: xxx
Hostname: 373dc353753f
Decommission Status : Normal
Configured Capacity: 498971181056 (464.70 GB)
DFS Used: 78566 (76.72 KB)
Block Pool Used: 4096 (4 KB)
Non DFS Used: 219416452378 (204.35 GB)
DFS Remaining: 279292506112 (260.11 GB)
DFS Used%: 0.00%
DFS Remaining%: 55.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 28 20:15:13 UTC 2018
Last Block Report: Wed Feb 28 20:14:43 UTC 2018
Name: xxx
Hostname: 57a3a51fff80
Decommission Status : Normal
Configured Capacity: 498971181056 (464.70 GB)
DFS Used: 8192 (8 KB)
Block Pool Used: 4096 (4 KB)
Non DFS Used: 219416522752 (204.35 GB)
DFS Remaining: 279292506112 (260.11 GB)
DFS Used%: 0.00%
DFS Remaining%: 55.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 28 20:15:12 UTC 2018
Last Block Report: Wed Feb 28 20:14:42 UTC 2018
Name: xxx
Hostname: 0865db3359a9
Decommission Status : Normal
Configured Capacity: 498971181056 (464.70 GB)
DFS Used: 8192 (8 KB)
Block Pool Used: 4096 (4 KB)
Non DFS Used: 219382968320 (204.32 GB)
DFS Remaining: 279326060544 (260.14 GB)
DFS Used%: 0.00%
DFS Remaining%: 55.98%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 28 20:15:12 UTC 2018
Last Block Report: Wed Feb 28 20:14:42 UTC 2018
Name: xxx
Hostname: e7d4bc7c9353
Decommission Status : Normal
Configured Capacity: 498971181056 (464.70 GB)
DFS Used: 8192 (8 KB)
Block Pool Used: 4096 (4 KB)
Non DFS Used: 219416522752 (204.35 GB)
DFS Remaining: 279292506112 (260.11 GB)
DFS Used%: 0.00%
DFS Remaining%: 55.97%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Feb 28 20:15:12 UTC 2018
Last Block Report: Wed Feb 28 20:14:42 UTC 2018
{code}
> Add blockpool used for DFSAdmin Report command
> ---
>
> Key: HDFS-13207
> URL: https://issues.apache.org/jira/browse/HDFS-13207
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13207.00.patch
>
>
> Add block pool used for dfsadmin -report command.
> This will be helpful on a federated cluster to know DFS used by a particular
> namespace.
>
> On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[
https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381046#comment-16381046
]
genericqa commented on HDFS-13207:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
31s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13207 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912479/HDFS-13207.00.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 5b6a9b0a4da3 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3100903 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23246/testReport/ |
| Max. process+thread count | 319 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23246/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add blockpool use
[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[ https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381048#comment-16381048 ] Xiaoyu Yao commented on HDFS-13207: --- Thanks [~bharatviswa] for reporting the issue and posting the patch. I just have few questions: 1. In the patch, we added the block pool used per datanode. Should we add "block pool used" to the summary portion before the "Live datanodes" as well? 2. If there are multiple namespaces and block pools, is it possible to separate the usage of different block pool? 3. Should we default to turn this off and show the "block pool used" only when federation is configured or a CLI switch like "-blockPoolUsed" is provided? > Add blockpool used for DFSAdmin Report command > --- > > Key: HDFS-13207 > URL: https://issues.apache.org/jira/browse/HDFS-13207 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13207.00.patch > > > Add block pool used for dfsadmin -report command. > This will be helpful on a federated cluster to know DFS used by a particular > namespace. > > On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381057#comment-16381057 ] Íñigo Goiri commented on HDFS-13204: A nameservice would also have those 4 states: * Active, when one namenode in the nameservice is active and no other state (e.g., safe mode) * Standby, when all the namenodes are in standby * Safe mode, when the active namenode is in safe mode * Unavailable, when all the namenodes are unavailable For the Datanode tab, we should exactly what the Namenode Datanode tab shows. Is this broken? > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[
https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381063#comment-16381063
]
genericqa commented on HDFS-13102:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 39s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 14 unchanged - 0 fixed = 15 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}159m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-13102 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12912468/HDFS-13102.009.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0d36005dacba 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / f20e10b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23245/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23245/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23245/te
[jira] [Commented] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[ https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381068#comment-16381068 ] Tsz Wo Nicholas Sze commented on HDFS-13102: +1 the 009 patch looks good. For the checkstyle warning on a redundant final, I will just remove it before committing the patch. > Implement SnapshotSkipList class to store Multi level DirectoryDiffs > > > Key: HDFS-13102 > URL: https://issues.apache.org/jira/browse/HDFS-13102 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Attachments: HDFS-13102.001.patch, HDFS-13102.002.patch, > HDFS-13102.003.patch, HDFS-13102.004.patch, HDFS-13102.005.patch, > HDFS-13102.006.patch, HDFS-13102.007.patch, HDFS-13102.008.patch, > HDFS-13102.009.patch > > > HDFS-11225 explains an issue where deletion of older snapshots can take a > very long time in case the no of snapshot diffs is quite large for > directories. For any directory under a snapshot, to construct the children > list , it needs to combine all the diffs from that particular snapshot to the > last snapshotDiff record and reverseApply to the current children list of the > directory on live fs. This can take a significant time if the no of snapshot > diffs are quite large and changes per diff is significant. > This Jira proposes to store the Directory diffs in a SnapshotSkip list, where > we store multi level DirectoryDiffs. At each level, the Directory Diff will > be cumulative diff of k snapshot diffs, > where k is the level of a node in the list. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[ https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-13102: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) I have committed this. Thanks, Shash! > Implement SnapshotSkipList class to store Multi level DirectoryDiffs > > > Key: HDFS-13102 > URL: https://issues.apache.org/jira/browse/HDFS-13102 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13102.001.patch, HDFS-13102.002.patch, > HDFS-13102.003.patch, HDFS-13102.004.patch, HDFS-13102.005.patch, > HDFS-13102.006.patch, HDFS-13102.007.patch, HDFS-13102.008.patch, > HDFS-13102.009.patch > > > HDFS-11225 explains an issue where deletion of older snapshots can take a > very long time in case the no of snapshot diffs is quite large for > directories. For any directory under a snapshot, to construct the children > list , it needs to combine all the diffs from that particular snapshot to the > last snapshotDiff record and reverseApply to the current children list of the > directory on live fs. This can take a significant time if the no of snapshot > diffs are quite large and changes per diff is significant. > This Jira proposes to store the Directory diffs in a SnapshotSkip list, where > we store multi level DirectoryDiffs. At each level, the Directory Diff will > be cumulative diff of k snapshot diffs, > where k is the level of a node in the list. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[ https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-13102: --- Attachment: HDFS-13102.009_committed.patch > Implement SnapshotSkipList class to store Multi level DirectoryDiffs > > > Key: HDFS-13102 > URL: https://issues.apache.org/jira/browse/HDFS-13102 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13102.001.patch, HDFS-13102.002.patch, > HDFS-13102.003.patch, HDFS-13102.004.patch, HDFS-13102.005.patch, > HDFS-13102.006.patch, HDFS-13102.007.patch, HDFS-13102.008.patch, > HDFS-13102.009.patch, HDFS-13102.009_committed.patch > > > HDFS-11225 explains an issue where deletion of older snapshots can take a > very long time in case the no of snapshot diffs is quite large for > directories. For any directory under a snapshot, to construct the children > list , it needs to combine all the diffs from that particular snapshot to the > last snapshotDiff record and reverseApply to the current children list of the > directory on live fs. This can take a significant time if the no of snapshot > diffs are quite large and changes per diff is significant. > This Jira proposes to store the Directory diffs in a SnapshotSkip list, where > we store multi level DirectoryDiffs. At each level, the Directory Diff will > be cumulative diff of k snapshot diffs, > where k is the level of a node in the list. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13143) SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff calculation happens between a snapshot and the current tree
[ https://issues.apache.org/jira/browse/HDFS-13143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-13143: --- Attachment: HDFS-13143.003_committed.patch > SnapshotDiff - snapshotDiffReport might be inconsistent if the snapshotDiff > calculation happens between a snapshot and the current tree > --- > > Key: HDFS-13143 > URL: https://issues.apache.org/jira/browse/HDFS-13143 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.1.0 > > Attachments: HDFS-13143.001.patch, HDFS-13143.002.patch, > HDFS-13143.003.patch, HDFS-13143.003_committed.patch > > > HDFS-12594 introduced an iterative approach for computing snapshotDiffs over > multiple rpc calls. The iterative approach depends upon the startPath (path > of the directory with respect to the snapshottable root) and the size of the > createdList (0 in case the startPath refers a file) to exactly determine form > where in each iteration the calculation has to start. > > In case of the diff computation between a snapshot and current tree(if any of > the snapshot names specified in the getSnapshotDiffReport call is null or > empty), the last SnapshotDiff associated with directory/file might change > owing to changes in the current tree in between the rpc calls in the absence > of a global fsn lock. This might result in consistencies in the > snapshotDiffReport. > In case the snapshotDiffReport computation needs to be done between the > current tree and a snapshot, we should fall back to non-iterative approach to > compute snapshotDiff. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13208) RBF: Mount path not available after ADD-REMOVE-ADD
Wei Yan created HDFS-13208:
--
Summary: RBF: Mount path not available after ADD-REMOVE-ADD
Key: HDFS-13208
URL: https://issues.apache.org/jira/browse/HDFS-13208
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Wei Yan
Assignee: Wei Yan
To reproduce this issue, run the following commands at Router 1:
{code:java}
$ hdfs dfsrouteradmin -add /test1 ns1 /ns1/test1
$ hdfs dfsrouteradmin -rm /test1
$ hdfs dfsrouteradmin -add /test1{code}
"hdfs dfs -ls hdfs://Router1:8020/test1" works well after step 1. But after
step 3 when we add /test1 back, Router 1 still returns "No such file or
directory".
But after step 3, when we run cmd "hdfs dfs -ls hdfs://Router2:8020/test1"
talking to another Router, it works well.
>From Router logs, I can see StateStoreZookeeperImpl and MountTableResolver are
>updated correctly and in time. Not find the root case yet, still looking into
>it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13102) Implement SnapshotSkipList class to store Multi level DirectoryDiffs
[ https://issues.apache.org/jira/browse/HDFS-13102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381092#comment-16381092 ] Hudson commented on HDFS-13102: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13741 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13741/]) HDFS-13102. Implement SnapshotSkipList class to store Multi level (szetszwo: rev 81d9446a92e3968234702b2981468a991c7cf8a0) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DiffList.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestDirectoryDiffList.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryDiffList.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DiffListByArrayList.java > Implement SnapshotSkipList class to store Multi level DirectoryDiffs > > > Key: HDFS-13102 > URL: https://issues.apache.org/jira/browse/HDFS-13102 > Project: Hadoop HDFS > Issue Type: Improvement > Components: snapshots >Reporter: Shashikant Banerjee >Assignee: Shashikant Banerjee >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13102.001.patch, HDFS-13102.002.patch, > HDFS-13102.003.patch, HDFS-13102.004.patch, HDFS-13102.005.patch, > HDFS-13102.006.patch, HDFS-13102.007.patch, HDFS-13102.008.patch, > HDFS-13102.009.patch, HDFS-13102.009_committed.patch > > > HDFS-11225 explains an issue where deletion of older snapshots can take a > very long time in case the no of snapshot diffs is quite large for > directories. For any directory under a snapshot, to construct the children > list , it needs to combine all the diffs from that particular snapshot to the > last snapshotDiff record and reverseApply to the current children list of the > directory on live fs. This can take a significant time if the no of snapshot > diffs are quite large and changes per diff is significant. > This Jira proposes to store the Directory diffs in a SnapshotSkip list, where > we store multi level DirectoryDiffs. At each level, the Directory Diff will > be cumulative diff of k snapshot diffs, > where k is the level of a node in the list. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13208) RBF: Mount path not available after ADD-REMOVE-ADD
[
https://issues.apache.org/jira/browse/HDFS-13208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-13208:
---
Description:
To reproduce this issue, run the following commands at Router 1:
{code:java}
$ hdfs dfsrouteradmin -add /test1 ns1 /ns1/test1
$ hdfs dfsrouteradmin -rm /test1
$ hdfs dfsrouteradmin -add /test1 ns1 /ns1/test1{code}
"hdfs dfs -ls hdfs://Router1:8020/test1" works well after step 1. After step 3
when we add /test1 back, Router 1 still returns "No such file or directory".
But after step 3, when we run cmd "hdfs dfs -ls hdfs://Router2:8020/test1"
talking to another Router, it works well.
>From Router logs, I can see StateStoreZookeeperImpl and MountTableResolver are
>updated correctly and in time. Not find the root case yet, still looking into
>it.
was:
To reproduce this issue, run the following commands at Router 1:
{code:java}
$ hdfs dfsrouteradmin -add /test1 ns1 /ns1/test1
$ hdfs dfsrouteradmin -rm /test1
$ hdfs dfsrouteradmin -add /test1{code}
"hdfs dfs -ls hdfs://Router1:8020/test1" works well after step 1. But after
step 3 when we add /test1 back, Router 1 still returns "No such file or
directory".
But after step 3, when we run cmd "hdfs dfs -ls hdfs://Router2:8020/test1"
talking to another Router, it works well.
>From Router logs, I can see StateStoreZookeeperImpl and MountTableResolver are
>updated correctly and in time. Not find the root case yet, still looking into
>it.
> RBF: Mount path not available after ADD-REMOVE-ADD
> --
>
> Key: HDFS-13208
> URL: https://issues.apache.org/jira/browse/HDFS-13208
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Critical
>
> To reproduce this issue, run the following commands at Router 1:
> {code:java}
> $ hdfs dfsrouteradmin -add /test1 ns1 /ns1/test1
> $ hdfs dfsrouteradmin -rm /test1
> $ hdfs dfsrouteradmin -add /test1 ns1 /ns1/test1{code}
> "hdfs dfs -ls hdfs://Router1:8020/test1" works well after step 1. After step
> 3 when we add /test1 back, Router 1 still returns "No such file or
> directory".
> But after step 3, when we run cmd "hdfs dfs -ls hdfs://Router2:8020/test1"
> talking to another Router, it works well.
> From Router logs, I can see StateStoreZookeeperImpl and MountTableResolver
> are updated correctly and in time. Not find the root case yet, still looking
> into it.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13171) Handle Deletion of nodes in SnasphotSkipList
[
https://issues.apache.org/jira/browse/HDFS-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381137#comment-16381137
]
Tsz Wo Nicholas Sze commented on HDFS-13171:
- Question: What if nodePath[level] == head? The code does nothing. Would the
remove(..) work for removing the first node?
{code}
if (nodePath[level] != head) {
...
}
if (nodeLevel == headLevel) {
while (headLevel > 0 && head.getSkipNode(headLevel) == null) {
headLevel--;
}
head.skipDiffList = head.skipDiffList.subList(0, headLevel + 1);
}
{code}
It seems that the remove code never update head. If it is true, there is some
bugs.
- remove(..) and addLast(..) both need to find the previous nodes. Let's add a
findPreviousNodes method as below
{code}
SkipListNode[] findPreviousNodes(SkipListNode node, int nodeLevel) {
final SkipListNode[] nodePath = new SkipListNode[nodeLevel + 1];
SkipListNode cur = head;
final int headLevel = head.level();
for (int level = headLevel < nodeLevel ? headLevel : nodeLevel;
level >= 0; level--) {
while (cur.getSkipNode(level) != node) {
cur = cur.getSkipNode(level);
}
nodePath[level] = cur;
}
for (int level = headLevel + 1; level <= nodeLevel; level++) {
nodePath[level] = head;
}
return nodePath;
}
{code}
> Handle Deletion of nodes in SnasphotSkipList
>
>
> Key: HDFS-13171
> URL: https://issues.apache.org/jira/browse/HDFS-13171
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDFS-13171.000.patch
>
>
> This Jira will handle deletion of skipListNodes from DirectoryDiffList . If a
> node has multiple levels, the list needs to be balanced .If the node is uni
> level, no balancing of the list is required.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[
https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381198#comment-16381198
]
Bharat Viswanadham commented on HDFS-13207:
---
Hi [~xyao]
Thank you for the comments.
{quote} 1. In the patch, we added the block pool used per datanode. Should we
add "block pool used" to the summary portion before the "Live datanodes" as
well?
{quote}
I think, we can add blockPoolUsed, which will show cumulative blockPoolUsed for
this namespace.
{quote}2. If there are multiple namespaces and block pools, is it possible to
separate the usage of different block pool?
{quote}
As DFSAdmin report command is made from one namespace, it will show
blockPoolused for that namespace. As DatanodeDescriptor maintains blockpoolused
for the namespace it belongs.
{quote}3. Should we default to turn this off and show the "block pool used"
only when federation is configured or a CLI switch like "-blockPoolUsed" is
provided?
{quote}
I think it is a good idea, will do that. I will try to use a switch
blockPoolUsed for dfsadmin -report command to do this.
> Add blockpool used for DFSAdmin Report command
> ---
>
> Key: HDFS-13207
> URL: https://issues.apache.org/jira/browse/HDFS-13207
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HDFS-13207.00.patch
>
>
> Add block pool used for dfsadmin -report command.
> This will be helpful on a federated cluster to know DFS used by a particular
> namespace.
>
> On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13209) DistributedFileSystem.create should allow an option to provide StoragePolicy
Jean-Marc Spaggiari created HDFS-13209: -- Summary: DistributedFileSystem.create should allow an option to provide StoragePolicy Key: HDFS-13209 URL: https://issues.apache.org/jira/browse/HDFS-13209 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs Affects Versions: 3.0.0 Reporter: Jean-Marc Spaggiari DistributedFileSystem.create allows to get a FSDataOutputStream. The stored file and related blocks will used the directory based StoragePolicy. However, sometime, we might need to keep all files in the same directory (consistency constraint) but might want some of them on SSD (small, in my case) until they are processed and merger/removed. Then they will go on the default policy. When creating a file, it will be useful to have an option to specify a different StoragePolicy... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[ https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13207: -- Attachment: HDFS-13207.01.patch > Add blockpool used for DFSAdmin Report command > --- > > Key: HDFS-13207 > URL: https://issues.apache.org/jira/browse/HDFS-13207 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13207.00.patch, HDFS-13207.01.patch > > > Add block pool used for dfsadmin -report command. > This will be helpful on a federated cluster to know DFS used by a particular > namespace. > > On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13207) Add blockpool used for DFSAdmin Report command
[ https://issues.apache.org/jira/browse/HDFS-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381281#comment-16381281 ] Bharat Viswanadham commented on HDFS-13207: --- [~xyao] Attached the patch to address 1 and 3 comments. > Add blockpool used for DFSAdmin Report command > --- > > Key: HDFS-13207 > URL: https://issues.apache.org/jira/browse/HDFS-13207 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13207.00.patch, HDFS-13207.01.patch > > > Add block pool used for dfsadmin -report command. > This will be helpful on a federated cluster to know DFS used by a particular > namespace. > > On a non-federated cluster, blockPoolUsed and DFSUsed will be shown as same. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13114) CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
[
https://issues.apache.org/jira/browse/HDFS-13114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381301#comment-16381301
]
Hanisha Koneru commented on HDFS-13114:
---
Thank you [~xyao] for committing the patch and [~jojochuang] for the review.
> CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path
>
>
> Key: HDFS-13114
> URL: https://issues.apache.org/jira/browse/HDFS-13114
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, hdfs
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Fix For: 3.1.0, 3.0.2, 3.2.0
>
> Attachments: HDFS-13114.001.patch
>
>
> The {{crypto -reencryptZone -path }} command takes in a path
> argument. But when creating {{HdfsAdmin}} object, it takes the defaultFs
> instead of resolving from the path. This causes the following exception if
> the authority component in path does not match the authority of default Fs.
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://mycluster-node-1:8020/zone1
> IllegalArgumentException: Wrong FS: hdfs://mycluster-node-1:8020/zone1,
> expected: hdfs://ns1{code}
> {code:java}
> $ hdfs crypto -reencryptZone -start -path hdfs://ns2/zone2
> IllegalArgumentException: Wrong FS: hdfs://ns2/zone2, expected:
> hdfs://ns1{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381378#comment-16381378
]
maobaolong commented on HDFS-13195:
---
[~bharatviswa] Thank you for clarify for me very much. And what should i do
next?
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381385#comment-16381385 ] maobaolong commented on HDFS-13204: --- I think the icon can be replace to others, but it is another thing we should think about. The core problem i think is that we should not use the node state style class in the Subclusters page, we should use the new state style class and we can replace these icons to new icons to make much sense every time. > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13081) Datanode#checkSecureConfig should allow SASL and privileged HTTP
[ https://issues.apache.org/jira/browse/HDFS-13081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381389#comment-16381389 ] Ajay Kumar commented on HDFS-13081: --- [~xyao], [~jlowe], [~daryn] thanks for input and reviews. > Datanode#checkSecureConfig should allow SASL and privileged HTTP > > > Key: HDFS-13081 > URL: https://issues.apache.org/jira/browse/HDFS-13081 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, security >Affects Versions: 3.0.0 >Reporter: Xiaoyu Yao >Assignee: Ajay Kumar >Priority: Major > Fix For: 3.1.0, 3.0.2, 3.2.0 > > Attachments: HDFS-13081.000.patch, HDFS-13081.001.patch, > HDFS-13081.002.patch, HDFS-13081.003.patch, HDFS-13081.004.patch, > HDFS-13081.005.patch, HDFS-13081.006.patch > > > Datanode#checkSecureConfig currently check the following to determine if > secure datanode is enabled. > # The server has bound to privileged ports for RPC and HTTP via > SecureDataNodeStarter. > # The configuration enables SASL on DataTransferProtocol and HTTPS (no plain > HTTP) for the HTTP server. > Authentication of Datanode RPC server can be done either via SASL handshake > or JSVC/privilege RPC port. > This guarantees authentication of the datanode RPC server before a client > transmits a secret, such as a block access token. > Authentication of the HTTP server can also be done either via HTTPS/SSL or > JSVC/privilege HTTP port. This guarantees authentication of datandoe HTTP > server before a client transmits a secret, such as a delegation token. > This ticket is open to allow privileged HTTP as an alternative to HTTPS to > work with SASL based RPC protection. > > cc: [~cnauroth] , [~daryn], [~jnpandey] for additional feedback. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13204) RBF: Optimize name service safe mode icon
[ https://issues.apache.org/jira/browse/HDFS-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381403#comment-16381403 ] wangzhiyuan commented on HDFS-13204: Good catch, state icons should be stay the same, furthermore, should check if it is the only place need to switch the icon. > RBF: Optimize name service safe mode icon > - > > Key: HDFS-13204 > URL: https://issues.apache.org/jira/browse/HDFS-13204 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: liuhongtong >Priority: Minor > Attachments: HDFS-13204.001.patch, image-2018-02-28-18-33-09-972.png, > image-2018-02-28-18-33-47-661.png, image-2018-02-28-18-35-35-708.png > > > In federation health webpage, the safe mode icons of Subclusters and Routers > are inconsistent. > The safe mode icon of Subclusters may induce users the name service is > maintaining. > !image-2018-02-28-18-33-09-972.png! > The safe mode icon of Routers: > !image-2018-02-28-18-33-47-661.png! > In fact, if the name service is in safe mode, users can't do writing related > operations. So I think the safe mode icon in Subclusters should be modified, > which may be more reasonable. > !image-2018-02-28-18-35-35-708.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13208) RBF: Mount path not available after ADD-REMOVE-ADD
[
https://issues.apache.org/jira/browse/HDFS-13208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381431#comment-16381431
]
Yiqun Lin commented on HDFS-13208:
--
[~ywskycn], I tried to reproduce this issue, and did the test in UT
{{TestRouterAdminCLI}}. The result looked fine. My test code:
{code:java}
@Test
public void testRemoveMountTable() throws Exception {
String nsId = "ns0";
String src = "/test-rmmounttable";
String dest = "/rmmounttable";
String[] argv = new String[] {"-add", src, nsId, dest};
assertEquals(0, ToolRunner.run(admin, argv));
stateStore.loadCache(MountTableStoreImpl.class, true);
GetMountTableEntriesRequest getRequest = GetMountTableEntriesRequest
.newInstance(src);
GetMountTableEntriesResponse getResponse = client.getMountTableManager()
.getMountTableEntries(getRequest);
// ensure mount table added successfully
MountTable mountTable = getResponse.getEntries().get(0);
assertEquals(src, mountTable.getSourcePath());
argv = new String[] {"-rm", src};
assertEquals(0, ToolRunner.run(admin, argv));
stateStore.loadCache(MountTableStoreImpl.class, true);
getResponse = client.getMountTableManager()
.getMountTableEntries(getRequest);
assertEquals(0, getResponse.getEntries().size());
argv = new String[] {"-add", src, nsId, dest};
assertEquals(0, ToolRunner.run(admin, argv));
stateStore.loadCache(MountTableStoreImpl.class, true);
getResponse = client.getMountTableManager()
.getMountTableEntries(getRequest);
assertEquals(1, getResponse.getEntries().size());
}
{code}
> RBF: Mount path not available after ADD-REMOVE-ADD
> --
>
> Key: HDFS-13208
> URL: https://issues.apache.org/jira/browse/HDFS-13208
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Wei Yan
>Priority: Critical
>
> To reproduce this issue, run the following commands at Router 1:
> {code:java}
> $ hdfs dfsrouteradmin -add /test1 ns1 /ns1/test1
> $ hdfs dfsrouteradmin -rm /test1
> $ hdfs dfsrouteradmin -add /test1 ns1 /ns1/test1{code}
> "hdfs dfs -ls hdfs://Router1:8020/test1" works well after step 1. After step
> 3 when we add /test1 back, Router 1 still returns "No such file or
> directory".
> But after step 3, when we run cmd "hdfs dfs -ls hdfs://Router2:8020/test1"
> talking to another Router, it works well.
> From Router logs, I can see StateStoreZookeeperImpl and MountTableResolver
> are updated correctly and in time. Not find the root case yet, still looking
> into it.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11610) sun.net.spi.nameservice.NameService has moved to a new location
[
https://issues.apache.org/jira/browse/HDFS-11610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381437#comment-16381437
]
Takanobu Asanuma commented on HDFS-11610:
-
This patch still works fine. Yesterday, I confirmed that {{mvn clean install
-DskipTests}} passed with jdk8 and jdk9. +1 (non-binding).
> sun.net.spi.nameservice.NameService has moved to a new location
> ---
>
> Key: HDFS-11610
> URL: https://issues.apache.org/jira/browse/HDFS-11610
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-11610.001.patch
>
>
> sun.net.spi.nameservice.NameService was moved to
> java.net.InetAddress$NameService in Java 9. TestDFSClientFailover uses this
> class to spy nameservice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11610) sun.net.spi.nameservice.NameService has moved to a new location
[
https://issues.apache.org/jira/browse/HDFS-11610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381437#comment-16381437
]
Takanobu Asanuma edited comment on HDFS-11610 at 3/1/18 2:57 AM:
-
This patch still works fine. Yesterday, I confirmed that {{mvn clean install
-DskipTests}} passed HADOOP-12760, through jdk8 and jdk9. +1 (non-binding).
was (Author: tasanuma0829):
This patch still works fine. Yesterday, I confirmed that {{mvn clean install
-DskipTests}} passed with jdk8 and jdk9. +1 (non-binding).
> sun.net.spi.nameservice.NameService has moved to a new location
> ---
>
> Key: HDFS-11610
> URL: https://issues.apache.org/jira/browse/HDFS-11610
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-11610.001.patch
>
>
> sun.net.spi.nameservice.NameService was moved to
> java.net.InetAddress$NameService in Java 9. TestDFSClientFailover uses this
> class to spy nameservice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11610) sun.net.spi.nameservice.NameService has moved to a new location
[
https://issues.apache.org/jira/browse/HDFS-11610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381437#comment-16381437
]
Takanobu Asanuma edited comment on HDFS-11610 at 3/1/18 3:07 AM:
-
This patch still works fine. Yesterday, I confirmed that {{mvn clean install
-DskipTests}} passed with HADOOP-12760, through jdk8 and jdk9. +1 (non-binding).
was (Author: tasanuma0829):
This patch still works fine. Yesterday, I confirmed that {{mvn clean install
-DskipTests}} passed HADOOP-12760, through jdk8 and jdk9. +1 (non-binding).
> sun.net.spi.nameservice.NameService has moved to a new location
> ---
>
> Key: HDFS-11610
> URL: https://issues.apache.org/jira/browse/HDFS-11610
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HDFS-11610.001.patch
>
>
> sun.net.spi.nameservice.NameService was moved to
> java.net.InetAddress$NameService in Java 9. TestDFSClientFailover uses this
> class to spy nameservice.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13195) DataNode conf page cannot display the current value after reconfig
[
https://issues.apache.org/jira/browse/HDFS-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381461#comment-16381461
]
wangzhiyuan commented on HDFS-13195:
+1 LGTM
> DataNode conf page cannot display the current value after reconfig
> ---
>
> Key: HDFS-13195
> URL: https://issues.apache.org/jira/browse/HDFS-13195
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Minor
> Fix For: 2.7.1
>
> Attachments: HDFS-13195-branch-2.7.001.patch, HDFS-13195.001.patch
>
>
> Now the branch-2.7 support dfs.datanode.data.dir reconfig, but after i
> reconfig this key, the conf page's value is still the old config value.
> The reason is that:
> {code:java}
> public DatanodeHttpServer(final Configuration conf,
> final DataNode datanode,
> final ServerSocketChannel externalHttpChannel)
> throws IOException {
> this.conf = conf;
> Configuration confForInfoServer = new Configuration(conf);
> confForInfoServer.setInt(HttpServer2.HTTP_MAX_THREADS, 10);
> HttpServer2.Builder builder = new HttpServer2.Builder()
> .setName("datanode")
> .setConf(confForInfoServer)
> .setACL(new AccessControlList(conf.get(DFS_ADMIN, " ")))
> .hostName(getHostnameForSpnegoPrincipal(confForInfoServer))
> .addEndpoint(URI.create("http://localhost:0";))
> .setFindPort(true);
> this.infoServer = builder.build();
> {code}
> The confForInfoServer is a new configuration instance, while the dfsadmin
> reconfig the datanode's config, the config result cannot reflect to
> confForInfoServer, so we should use the datanode's conf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
