[jira] [Work logged] (HDFS-16314) Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16314?focusedWorklogId=681381&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681381 ] ASF GitHub Bot logged work on HDFS-16314: - Author: ASF GitHub Bot Created on: 15/Nov/21 09:03 Start Date: 15/Nov/21 09:03 Worklog Time Spent: 10m Work Description: haiyang1987 commented on pull request #3651: URL: https://github.com/apache/hadoop/pull/3651#issuecomment-968674241 @ferhui @tomscut I submitted some code. Can you help review. thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681381) Time Spent: 0.5h (was: 20m) > Support to make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > - > > Key: HDFS-16314 > URL: https://issues.apache.org/jira/browse/HDFS-16314 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Consider that make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > and rapid rollback in case this feature HDFS-16076 unexpected things happen > in production environment -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16320) Datanode retrieve slownode information from NameNode
[ https://issues.apache.org/jira/browse/HDFS-16320?focusedWorklogId=681411&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681411 ] ASF GitHub Bot logged work on HDFS-16320: - Author: ASF GitHub Bot Created on: 15/Nov/21 10:50 Start Date: 15/Nov/21 10:50 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3654: URL: https://github.com/apache/hadoop/pull/3654#issuecomment-968766994 @symious Thanks. Right now only namenode knows that which datanode is slow, and it will avoid to choose this node for the following requirement. But we have no ways to handle the writing pipeline. I think it's good idea, go ahead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681411) Time Spent: 40m (was: 0.5h) > Datanode retrieve slownode information from NameNode > > > Key: HDFS-16320 > URL: https://issues.apache.org/jira/browse/HDFS-16320 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The current information of slownode is reported by reportingNode, and stored > in NameNode. > This ticket is to let the slownode retrieve the information from NameNode, so > that it can do other performance improvement actions based on this > information. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16320) Datanode retrieve slownode information from NameNode
[ https://issues.apache.org/jira/browse/HDFS-16320?focusedWorklogId=681412&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681412 ] ASF GitHub Bot logged work on HDFS-16320: - Author: ASF GitHub Bot Created on: 15/Nov/21 10:52 Start Date: 15/Nov/21 10:52 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3654: URL: https://github.com/apache/hadoop/pull/3654#issuecomment-968767997 @aajisaka @Hexiaoqiao @jojochuang Would you give any advices? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681412) Time Spent: 50m (was: 40m) > Datanode retrieve slownode information from NameNode > > > Key: HDFS-16320 > URL: https://issues.apache.org/jira/browse/HDFS-16320 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > The current information of slownode is reported by reportingNode, and stored > in NameNode. > This ticket is to let the slownode retrieve the information from NameNode, so > that it can do other performance improvement actions based on this > information. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16313) RBF: Add metrics for each sub cluster
[ https://issues.apache.org/jira/browse/HDFS-16313?focusedWorklogId=681413&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681413 ] ASF GitHub Bot logged work on HDFS-16313: - Author: ASF GitHub Bot Created on: 15/Nov/21 10:58 Start Date: 15/Nov/21 10:58 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3638: URL: https://github.com/apache/hadoop/pull/3638#issuecomment-968774697 @goiri Do you have any other comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681413) Time Spent: 2h 50m (was: 2h 40m) > RBF: Add metrics for each sub cluster > - > > Key: HDFS-16313 > URL: https://issues.apache.org/jira/browse/HDFS-16313 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > Currently we have metrics to track the operations for Router to all > nameservices, like "FederationRPCMetrics", but we don't have metrics for > Router to each nameservices. > This ticket is to add metrics for each nameservice to better track the > performance of each sub cluster. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16322) The NameNode implementation of ClientProtocol.truncate(...) can cause data loss.
[ https://issues.apache.org/jira/browse/HDFS-16322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17443803#comment-17443803 ] nhaorand commented on HDFS-16322: - Thanks [~hexiaoqiao] and [~szetszwo] for your discussion. Considering the following case: # The initial length of the file is 100. # Client A sends a request of truncate() for the file. # NameNode executes truncate request, changes the file length to be 50 and responds to Client A. # Client B finds the length of the file is incorrect and requests append for the file. Then the file length becomes 100. # Client A retries the truncate because it does not achieve response. NameNode will re-execute it because no RetryCache for truncate request then respond to Client A. There is only one client sending truncate. However, from the view of client B, there are two truncate, which causes that new data are lost and the final length of the file is not 100. As a method to perform fault tolerance, retry should not make any individual client to realize that retry happens. But in this case, client B can realize that the client A issues a retry. > The NameNode implementation of ClientProtocol.truncate(...) can cause data > loss. > > > Key: HDFS-16322 > URL: https://issues.apache.org/jira/browse/HDFS-16322 > Project: Hadoop HDFS > Issue Type: Bug > Environment: The runtime environment is Ubuntu 18.04, Java 1.8.0_222 > and Apache Maven 3.6.0. > The bug can be reproduced by the the testMultipleTruncate() in the > attachment. First, replace the file TestFileTruncate.java under the directory > "hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/" > with the attachment. Then run "mvn test > -Dtest=org.apache.hadoop.hdfs.server.namenode.TestFileTruncate#testMultipleTruncate" > to run the testcase. Finally the "assertFileLength(p, n+newLength)" at 199 > line of TestFileTruncate.java will abort. Because the retry of truncate() > changes the file size and cause data loss. >Reporter: nhaorand >Priority: Major > Attachments: TestFileTruncate.java > > > The NameNode implementation of ClientProtocol.truncate(...) can cause data > loss. If dfsclient drops the first response of a truncate RPC call, the retry > by retry cache will truncate the file again and cause data loss. > HDFS-7926 avoids repeated execution of truncate(...) by checking if the file > is already being truncated with the same length. However, under concurrency, > after the first execution of truncate(...), concurrent requests from other > clients may append new data and change the file length. When truncate(...) is > retried after that, it will find the file has not been truncated with the > same length and truncate it again, which causes data loss. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16322) The NameNode implementation of ClientProtocol.truncate(...) can cause data loss.
[ https://issues.apache.org/jira/browse/HDFS-16322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17443803#comment-17443803 ] nhaorand edited comment on HDFS-16322 at 11/15/21, 12:45 PM: - Thanks [~hexiaoqiao] and [~szetszwo] for your discussion. Considering the following case: # The initial length of the file is 100. # Client A sends a request of truncate() for the file. # NameNode executes truncate request, changes the file length to be 50 and responds to Client A. # Client B finds the length of the file is incorrect and requests append for the file. Then the file length becomes 100. # Client A retries the truncate because it does not achieve response. NameNode will re-execute it because no RetryCache for truncate request then respond to Client A. There is only one client sending truncate. However, from the view of client B, there are two truncate, which causes that new data are lost and the final length of the file is not 100. As a method to perform fault tolerance, retry should not make any individual client to realize that retry happens. But in this case, client B can realize that the client A issues a retry. Therefore, truncate() should be fixed. was (Author: JIRAUSER280052): Thanks [~hexiaoqiao] and [~szetszwo] for your discussion. Considering the following case: # The initial length of the file is 100. # Client A sends a request of truncate() for the file. # NameNode executes truncate request, changes the file length to be 50 and responds to Client A. # Client B finds the length of the file is incorrect and requests append for the file. Then the file length becomes 100. # Client A retries the truncate because it does not achieve response. NameNode will re-execute it because no RetryCache for truncate request then respond to Client A. There is only one client sending truncate. However, from the view of client B, there are two truncate, which causes that new data are lost and the final length of the file is not 100. As a method to perform fault tolerance, retry should not make any individual client to realize that retry happens. But in this case, client B can realize that the client A issues a retry. > The NameNode implementation of ClientProtocol.truncate(...) can cause data > loss. > > > Key: HDFS-16322 > URL: https://issues.apache.org/jira/browse/HDFS-16322 > Project: Hadoop HDFS > Issue Type: Bug > Environment: The runtime environment is Ubuntu 18.04, Java 1.8.0_222 > and Apache Maven 3.6.0. > The bug can be reproduced by the the testMultipleTruncate() in the > attachment. First, replace the file TestFileTruncate.java under the directory > "hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/" > with the attachment. Then run "mvn test > -Dtest=org.apache.hadoop.hdfs.server.namenode.TestFileTruncate#testMultipleTruncate" > to run the testcase. Finally the "assertFileLength(p, n+newLength)" at 199 > line of TestFileTruncate.java will abort. Because the retry of truncate() > changes the file size and cause data loss. >Reporter: nhaorand >Priority: Major > Attachments: TestFileTruncate.java > > > The NameNode implementation of ClientProtocol.truncate(...) can cause data > loss. If dfsclient drops the first response of a truncate RPC call, the retry > by retry cache will truncate the file again and cause data loss. > HDFS-7926 avoids repeated execution of truncate(...) by checking if the file > is already being truncated with the same length. However, under concurrency, > after the first execution of truncate(...), concurrent requests from other > clients may append new data and change the file length. When truncate(...) is > retried after that, it will find the file has not been truncated with the > same length and truncate it again, which causes data loss. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16324) fix error log in BlockManagerSafeMode
[ https://issues.apache.org/jira/browse/HDFS-16324?focusedWorklogId=681466&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681466 ] ASF GitHub Bot logged work on HDFS-16324: - Author: ASF GitHub Bot Created on: 15/Nov/21 13:05 Start Date: 15/Nov/21 13:05 Worklog Time Spent: 10m Work Description: GuoPhilipse opened a new pull request #3661: URL: https://github.com/apache/hadoop/pull/3661 ### Description of PR if `recheckInterval` was set as invalid value, there will be warning log output, but the message seems not that proper ,we can improve it. ### How was this patch tested? No need test cases, just update warning log. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681466) Remaining Estimate: 0h Time Spent: 10m > fix error log in BlockManagerSafeMode > - > > Key: HDFS-16324 > URL: https://issues.apache.org/jira/browse/HDFS-16324 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > if `recheckInterval` was set as invalid value, there will be warning log > output, but the message seems not that proper ,we can improve it. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16324) fix error log in BlockManagerSafeMode
guo created HDFS-16324: -- Summary: fix error log in BlockManagerSafeMode Key: HDFS-16324 URL: https://issues.apache.org/jira/browse/HDFS-16324 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.3.1 Reporter: guo if `recheckInterval` was set as invalid value, there will be warning log output, but the message seems not that proper ,we can improve it. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16324) fix error log in BlockManagerSafeMode
[ https://issues.apache.org/jira/browse/HDFS-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16324: -- Labels: pull-request-available (was: ) > fix error log in BlockManagerSafeMode > - > > Key: HDFS-16324 > URL: https://issues.apache.org/jira/browse/HDFS-16324 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > if `recheckInterval` was set as invalid value, there will be warning log > output, but the message seems not that proper ,we can improve it. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16323) DatanodeHttpServer doesn't require handler state map while retrieving filter handlers
[
https://issues.apache.org/jira/browse/HDFS-16323?focusedWorklogId=681502&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681502
]
ASF GitHub Bot logged work on HDFS-16323:
-
Author: ASF GitHub Bot
Created on: 15/Nov/21 13:47
Start Date: 15/Nov/21 13:47
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3659:
URL: https://github.com/apache/hadoop/pull/3659#issuecomment-968926593
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 347m 29s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3659/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 455m 7s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3659/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3659 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux c29d9d04303d 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 81c1a76ea3b24bf05c7ab826ba133d07b830a068 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job
[jira] [Created] (HDFS-16325) some time delay (0.3s) for swebhdfs + kerberos + observer setting.
Chuan-Heng Hsiao created HDFS-16325: --- Summary: some time delay (0.3s) for swebhdfs + kerberos + observer setting. Key: HDFS-16325 URL: https://issues.apache.org/jira/browse/HDFS-16325 Project: Hadoop HDFS Issue Type: Bug Components: ha, webhdfs Affects Versions: 3.3.1 Environment: ubuntu 20.04 hadoop 3.3.1 openjdk-8 Reporter: Chuan-Heng Hsiao Settings: 1 master namenode (A), 1 standby namenode (B), 1 observer namenode (C). following [https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html] except that dfs.client.failover.observer.auto-msync-period. is set to -1 (not auto -msync) uable to do curl - - negotiate -u ':' 'https://:/webhdfs/v1/...' because it seems like due to the following issue: https://issues.apache.org/jira/browse/HDFS-14443 using curl --negotiate -u ':' 'https://:/webhdfs/v1/...' can successfully get 307 redirect with the corresponding Location. but got token (token for xxx HDFS_DELEGATION_TOKEN owner=xxx renewer=xxx masterKeyID=ooo) can't be found in cache" if redirect the url within 300ms. Not issue if waiting for more than 300ms and then do the redirect. No issue if changing (C) to Standby (no observers) (and redirect within 10 ms) -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16322) The NameNode implementation of ClientProtocol.truncate(...) can cause data loss.
[
https://issues.apache.org/jira/browse/HDFS-16322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17444033#comment-17444033
]
Konstantin Shvachko commented on HDFS-16322:
Is it specific to truncate?
Same thing should happen with {{mkdir()}}. Client A creates a directory, client
B deletes it, then client A retries the create. Same with {{setPermission()}}?
>From NN perspective the two calls from client A are different calls. Since NN
>responded to the first call from client A, it treats the retry as the second
>call.
> The NameNode implementation of ClientProtocol.truncate(...) can cause data
> loss.
>
>
> Key: HDFS-16322
> URL: https://issues.apache.org/jira/browse/HDFS-16322
> Project: Hadoop HDFS
> Issue Type: Bug
> Environment: The runtime environment is Ubuntu 18.04, Java 1.8.0_222
> and Apache Maven 3.6.0.
> The bug can be reproduced by the the testMultipleTruncate() in the
> attachment. First, replace the file TestFileTruncate.java under the directory
> "hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/"
> with the attachment. Then run "mvn test
> -Dtest=org.apache.hadoop.hdfs.server.namenode.TestFileTruncate#testMultipleTruncate"
> to run the testcase. Finally the "assertFileLength(p, n+newLength)" at 199
> line of TestFileTruncate.java will abort. Because the retry of truncate()
> changes the file size and cause data loss.
>Reporter: nhaorand
>Priority: Major
> Attachments: TestFileTruncate.java
>
>
> The NameNode implementation of ClientProtocol.truncate(...) can cause data
> loss. If dfsclient drops the first response of a truncate RPC call, the retry
> by retry cache will truncate the file again and cause data loss.
> HDFS-7926 avoids repeated execution of truncate(...) by checking if the file
> is already being truncated with the same length. However, under concurrency,
> after the first execution of truncate(...), concurrent requests from other
> clients may append new data and change the file length. When truncate(...) is
> retried after that, it will find the file has not been truncated with the
> same length and truncate it again, which causes data loss.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16324) fix error log in BlockManagerSafeMode
[
https://issues.apache.org/jira/browse/HDFS-16324?focusedWorklogId=681668&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681668
]
ASF GitHub Bot logged work on HDFS-16324:
-
Author: ASF GitHub Bot
Created on: 15/Nov/21 19:00
Start Date: 15/Nov/21 19:00
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3661:
URL: https://github.com/apache/hadoop/pull/3661#issuecomment-969219671
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 26s | | trunk passed |
| +1 :green_heart: | compile | 5m 12s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 4m 52s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 10s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 2s | | the patch passed |
| +1 :green_heart: | compile | 5m 12s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 5m 12s | | the patch passed |
| +1 :green_heart: | compile | 4m 40s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 4m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 4s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/1/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 133 new + 34 unchanged - 0 fixed =
167 total (was 34) |
| +1 :green_heart: | mvnsite | 2m 3s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 50s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 5m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 25s | | hadoop-hdfs-client in the patch
passed. |
| -1 :x: | unit | 226m 6s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 352m 51s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3661 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 1500bb72bf63 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 93e21da8e1ee16ce52580927e2c40d2b43
[jira] [Created] (HDFS-16326) Simplify the code for DiskBalancer
tomscut created HDFS-16326: -- Summary: Simplify the code for DiskBalancer Key: HDFS-16326 URL: https://issues.apache.org/jira/browse/HDFS-16326 Project: Hadoop HDFS Issue Type: Wish Reporter: tomscut Assignee: tomscut Simplify the code for DiskBalancer. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16326) Simplify the code for DiskBalancer
[ https://issues.apache.org/jira/browse/HDFS-16326?focusedWorklogId=681774&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681774 ] ASF GitHub Bot logged work on HDFS-16326: - Author: ASF GitHub Bot Created on: 16/Nov/21 00:47 Start Date: 16/Nov/21 00:47 Worklog Time Spent: 10m Work Description: tomscut opened a new pull request #3663: URL: https://github.com/apache/hadoop/pull/3663 JIRA: [HDFS-16326](https://issues.apache.org/jira/browse/HDFS-16326). Simplify the code for DiskBalancer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681774) Remaining Estimate: 0h Time Spent: 10m > Simplify the code for DiskBalancer > -- > > Key: HDFS-16326 > URL: https://issues.apache.org/jira/browse/HDFS-16326 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Simplify the code for DiskBalancer. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16326) Simplify the code for DiskBalancer
[ https://issues.apache.org/jira/browse/HDFS-16326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16326: -- Labels: pull-request-available (was: ) > Simplify the code for DiskBalancer > -- > > Key: HDFS-16326 > URL: https://issues.apache.org/jira/browse/HDFS-16326 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Simplify the code for DiskBalancer. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16314) Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16314?focusedWorklogId=681784&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681784
]
ASF GitHub Bot logged work on HDFS-16314:
-
Author: ASF GitHub Bot
Created on: 16/Nov/21 01:38
Start Date: 16/Nov/21 01:38
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #3651:
URL: https://github.com/apache/hadoop/pull/3651#discussion_r749833779
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
##
@@ -2390,15 +2394,24 @@ String reconfigureParallelLoad(String newVal) {
String reconfigureSlowNodesParameters(final DatanodeManager datanodeManager,
final String property, final String newVal) throws
ReconfigurationException {
+BlockManager bm = namesystem.getBlockManager();
namesystem.writeLock();
boolean enable;
try {
- if (newVal == null) {
-enable = DFS_NAMENODE_AVOID_SLOW_DATANODE_FOR_READ_DEFAULT;
+ if (property.equals(DFS_NAMENODE_AVOID_SLOW_DATANODE_FOR_READ_KEY)) {
+enable = (newVal == null ?
DFS_NAMENODE_AVOID_SLOW_DATANODE_FOR_READ_DEFAULT :
+Boolean.parseBoolean(newVal));
+datanodeManager.setAvoidSlowDataNodesForReadEnabled(enable);
+ } else if (property.equals(
+DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_KEY))
{
+enable = (newVal == null ?
+
DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_DEFAULT :
+Boolean.parseBoolean(newVal));
+bm.setExculeSlowDataNodesForWriteEnabled(enable);
} else {
-enable = Boolean.parseBoolean(newVal);
+throw new IllegalArgumentException("Unexpected property " +
+property + "in reconfReplicationParameters");
Review comment:
There is a space missing and the method name needs to be changed to
`reconfigureSlowNodesParameters`. BTW, please add a space in line `2237`.
Thanks.
The other changes look good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 681784)
Time Spent: 40m (was: 0.5h)
> Support to make
> dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
> -
>
> Key: HDFS-16314
> URL: https://issues.apache.org/jira/browse/HDFS-16314
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Consider that make
> dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
> and rapid rollback in case this feature HDFS-16076 unexpected things happen
> in production environment
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16313) RBF: Add metrics for each sub cluster
[ https://issues.apache.org/jira/browse/HDFS-16313?focusedWorklogId=681786&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681786 ] ASF GitHub Bot logged work on HDFS-16313: - Author: ASF GitHub Bot Created on: 16/Nov/21 01:47 Start Date: 16/Nov/21 01:47 Worklog Time Spent: 10m Work Description: ferhui merged pull request #3638: URL: https://github.com/apache/hadoop/pull/3638 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681786) Time Spent: 3h (was: 2h 50m) > RBF: Add metrics for each sub cluster > - > > Key: HDFS-16313 > URL: https://issues.apache.org/jira/browse/HDFS-16313 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > Currently we have metrics to track the operations for Router to all > nameservices, like "FederationRPCMetrics", but we don't have metrics for > Router to each nameservices. > This ticket is to add metrics for each nameservice to better track the > performance of each sub cluster. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16313) RBF: Add metrics for each sub cluster
[ https://issues.apache.org/jira/browse/HDFS-16313?focusedWorklogId=681787&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681787 ] ASF GitHub Bot logged work on HDFS-16313: - Author: ASF GitHub Bot Created on: 16/Nov/21 01:48 Start Date: 16/Nov/21 01:48 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3638: URL: https://github.com/apache/hadoop/pull/3638#issuecomment-969653176 @symious Thanks for contribution. @goiri Thanks for review! Merged to trunk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681787) Time Spent: 3h 10m (was: 3h) > RBF: Add metrics for each sub cluster > - > > Key: HDFS-16313 > URL: https://issues.apache.org/jira/browse/HDFS-16313 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 3h 10m > Remaining Estimate: 0h > > Currently we have metrics to track the operations for Router to all > nameservices, like "FederationRPCMetrics", but we don't have metrics for > Router to each nameservices. > This ticket is to add metrics for each nameservice to better track the > performance of each sub cluster. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16327) Change dfs.namenode.max.slowpeer.collect.nodes to a proportional value
tomscut created HDFS-16327: -- Summary: Change dfs.namenode.max.slowpeer.collect.nodes to a proportional value Key: HDFS-16327 URL: https://issues.apache.org/jira/browse/HDFS-16327 Project: Hadoop HDFS Issue Type: Wish Reporter: tomscut Assignee: tomscut Currently, dfs.namenode.max.slowpeer.collect.nodes is a fixed value, but often needs to be changed as the cluster size changes. We can change it to a scale value and make it reconfigurable. See [HDFS-15879|https://issues.apache.org/jira/browse/HDFS-15879]. And dfs.datanode.max.disks.to.report can be changed similarly. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16324) fix error log in BlockManagerSafeMode
[ https://issues.apache.org/jira/browse/HDFS-16324?focusedWorklogId=681789&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681789 ] ASF GitHub Bot logged work on HDFS-16324: - Author: ASF GitHub Bot Created on: 16/Nov/21 01:59 Start Date: 16/Nov/21 01:59 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3661: URL: https://github.com/apache/hadoop/pull/3661#issuecomment-969671740 @tomscut Could you kindly help verify, the test error seems not related with the patch. `[ERROR] testSetRepIncWithUnderReplicatedBlocks(org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks) Time elapsed: 120.023 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.shell.SetReplication.waitForReplication(SetReplication.java:137) at org.apache.hadoop.fs.shell.SetReplication.processArguments(SetReplication.java:78) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121) at org.apache.hadoop.fs.shell.Command.run(Command.java:179) at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) at org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks(TestUnderReplicatedBlocks.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748)` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681789) Time Spent: 0.5h (was: 20m) > fix error log in BlockManagerSafeMode > - > > Key: HDFS-16324 > URL: https://issues.apache.org/jira/browse/HDFS-16324 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > if `recheckInterval` was set as invalid value, there will be warning log > output, but the message seems not that proper ,we can improve it. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16320) Datanode retrieve slownode information from NameNode
[ https://issues.apache.org/jira/browse/HDFS-16320?focusedWorklogId=681790&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681790 ] ASF GitHub Bot logged work on HDFS-16320: - Author: ASF GitHub Bot Created on: 16/Nov/21 02:03 Start Date: 16/Nov/21 02:03 Worklog Time Spent: 10m Work Description: ferhui commented on pull request #3654: URL: https://github.com/apache/hadoop/pull/3654#issuecomment-969675032 @tasanuma @ayushtkn Would you also take a look? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681790) Time Spent: 1h (was: 50m) > Datanode retrieve slownode information from NameNode > > > Key: HDFS-16320 > URL: https://issues.apache.org/jira/browse/HDFS-16320 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Janus Chow >Assignee: Janus Chow >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The current information of slownode is reported by reportingNode, and stored > in NameNode. > This ticket is to let the slownode retrieve the information from NameNode, so > that it can do other performance improvement actions based on this > information. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16324) fix error log in BlockManagerSafeMode
[ https://issues.apache.org/jira/browse/HDFS-16324?focusedWorklogId=681791&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681791 ] ASF GitHub Bot logged work on HDFS-16324: - Author: ASF GitHub Bot Created on: 16/Nov/21 02:05 Start Date: 16/Nov/21 02:05 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3661: URL: https://github.com/apache/hadoop/pull/3661#issuecomment-969677031 > @tomscut Could you kindly help verify, the test error seems not related with the patch. `[ERROR] testSetRepIncWithUnderReplicatedBlocks(org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks) Time elapsed: 120.023 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.shell.SetReplication.waitForReplication(SetReplication.java:137) at org.apache.hadoop.fs.shell.SetReplication.processArguments(SetReplication.java:78) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121) at org.apache.hadoop.fs.shell.Command.run(Command.java:179) at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) at org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks(TestUnderReplicatedBlocks.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748)` Can you commit an empty commit to trigger the builder again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681791) Time Spent: 40m (was: 0.5h) > fix error log in BlockManagerSafeMode > - > > Key: HDFS-16324 > URL: https://issues.apache.org/jira/browse/HDFS-16324 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > if `recheckInterval` was set as invalid value, there will be warning log > output, but the message seems not that proper ,we can improve it. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16315) Add metrics related to Transfer and NativeCopy for DataNode
[ https://issues.apache.org/jira/browse/HDFS-16315?focusedWorklogId=681792&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681792 ] ASF GitHub Bot logged work on HDFS-16315: - Author: ASF GitHub Bot Created on: 16/Nov/21 02:19 Start Date: 16/Nov/21 02:19 Worklog Time Spent: 10m Work Description: tasanuma merged pull request #3643: URL: https://github.com/apache/hadoop/pull/3643 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681792) Time Spent: 2h 50m (was: 2h 40m) > Add metrics related to Transfer and NativeCopy for DataNode > --- > > Key: HDFS-16315 > URL: https://issues.apache.org/jira/browse/HDFS-16315 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-11-08-26-33-074.png > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Datanodes already have Read, Write, Sync and Flush metrics. We should add > NativeCopy and Transfer as well. > Here is a partial look after the change: > !image-2021-11-11-08-26-33-074.png|width=205,height=235! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16315) Add metrics related to Transfer and NativeCopy for DataNode
[ https://issues.apache.org/jira/browse/HDFS-16315?focusedWorklogId=681793&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681793 ] ASF GitHub Bot logged work on HDFS-16315: - Author: ASF GitHub Bot Created on: 16/Nov/21 02:20 Start Date: 16/Nov/21 02:20 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3643: URL: https://github.com/apache/hadoop/pull/3643#issuecomment-969691839 Merged. Thanks for your contribution, @tomscut, and thanks for reviewing it, @ferhui and @ayushtkn. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681793) Time Spent: 3h (was: 2h 50m) > Add metrics related to Transfer and NativeCopy for DataNode > --- > > Key: HDFS-16315 > URL: https://issues.apache.org/jira/browse/HDFS-16315 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-11-08-26-33-074.png > > Time Spent: 3h > Remaining Estimate: 0h > > Datanodes already have Read, Write, Sync and Flush metrics. We should add > NativeCopy and Transfer as well. > Here is a partial look after the change: > !image-2021-11-11-08-26-33-074.png|width=205,height=235! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16324) fix error log in BlockManagerSafeMode
[ https://issues.apache.org/jira/browse/HDFS-16324?focusedWorklogId=681795&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681795 ] ASF GitHub Bot logged work on HDFS-16324: - Author: ASF GitHub Bot Created on: 16/Nov/21 02:23 Start Date: 16/Nov/21 02:23 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3661: URL: https://github.com/apache/hadoop/pull/3661#issuecomment-969695706 > > @tomscut Could you kindly help verify, the test error seems not related with the patch. `[ERROR] testSetRepIncWithUnderReplicatedBlocks(org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks) Time elapsed: 120.023 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 12 milliseconds at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.fs.shell.SetReplication.waitForReplication(SetReplication.java:137) at org.apache.hadoop.fs.shell.SetReplication.processArguments(SetReplication.java:78) at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:121) at org.apache.hadoop.fs.shell.Command.run(Command.java:179) at org.apache.hadoop.fs.FsShell.run(FsShell.java:327) at org.apache.hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks.testSetRepIncWithUnderReplicatedBlocks(TestUnderReplicatedBlocks.java:80) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748)` > > Can you commit an empty commit to trigger the builder again? sure, have just triggered -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681795) Time Spent: 50m (was: 40m) > fix error log in BlockManagerSafeMode > - > > Key: HDFS-16324 > URL: https://issues.apache.org/jira/browse/HDFS-16324 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > if `recheckInterval` was set as invalid value, there will be warning log > output, but the message seems not that proper ,we can improve it. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16315) Add metrics related to Transfer and NativeCopy for DataNode
[ https://issues.apache.org/jira/browse/HDFS-16315?focusedWorklogId=681796&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681796 ] ASF GitHub Bot logged work on HDFS-16315: - Author: ASF GitHub Bot Created on: 16/Nov/21 02:36 Start Date: 16/Nov/21 02:36 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3643: URL: https://github.com/apache/hadoop/pull/3643#issuecomment-969716964 Thanks @tasanuma for the merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681796) Time Spent: 3h 10m (was: 3h) > Add metrics related to Transfer and NativeCopy for DataNode > --- > > Key: HDFS-16315 > URL: https://issues.apache.org/jira/browse/HDFS-16315 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-11-08-26-33-074.png > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Datanodes already have Read, Write, Sync and Flush metrics. We should add > NativeCopy and Transfer as well. > Here is a partial look after the change: > !image-2021-11-11-08-26-33-074.png|width=205,height=235! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16315) Add metrics related to Transfer and NativeCopy for DataNode
[ https://issues.apache.org/jira/browse/HDFS-16315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma resolved HDFS-16315. - Fix Version/s: 3.4.0 Resolution: Fixed > Add metrics related to Transfer and NativeCopy for DataNode > --- > > Key: HDFS-16315 > URL: https://issues.apache.org/jira/browse/HDFS-16315 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2021-11-11-08-26-33-074.png > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Datanodes already have Read, Write, Sync and Flush metrics. We should add > NativeCopy and Transfer as well. > Here is a partial look after the change: > !image-2021-11-11-08-26-33-074.png|width=205,height=235! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15457) TestFsDatasetImpl fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-15457:
Fix Version/s: 3.3.2
> TestFsDatasetImpl fails intermittently
> --
>
> Key: HDFS-15457
> URL: https://issues.apache.org/jira/browse/HDFS-15457
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.3.2, 3.2.4
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {{TestFsDatasetImpl}} fails intermittently on hadoop trunk.
> {code:bash}
> [ERROR] Tests run: 21, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 30.104 s <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> [ERROR]
> testReadLockEnabledByDefault(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 0.128 s <<< FAILURE!
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testReadLockEnabledByDefault(TestFsDatasetImpl.java:237)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15457) TestFsDatasetImpl fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-15457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17444244#comment-17444244
]
Takanobu Asanuma commented on HDFS-15457:
-
Cherry-picked to branch-3.3.
> TestFsDatasetImpl fails intermittently
> --
>
> Key: HDFS-15457
> URL: https://issues.apache.org/jira/browse/HDFS-15457
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.3, 3.2.4
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> {{TestFsDatasetImpl}} fails intermittently on hadoop trunk.
> {code:bash}
> [ERROR] Tests run: 21, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 30.104 s <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
> [ERROR]
> testReadLockEnabledByDefault(org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl)
> Time elapsed: 0.128 s <<< FAILURE!
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl.testReadLockEnabledByDefault(TestFsDatasetImpl.java:237)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16314) Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16314?focusedWorklogId=681803&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681803
]
ASF GitHub Bot logged work on HDFS-16314:
-
Author: ASF GitHub Bot
Created on: 16/Nov/21 03:13
Start Date: 16/Nov/21 03:13
Worklog Time Spent: 10m
Work Description: haiyang1987 commented on a change in pull request #3651:
URL: https://github.com/apache/hadoop/pull/3651#discussion_r749867573
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
##
@@ -2390,15 +2394,24 @@ String reconfigureParallelLoad(String newVal) {
String reconfigureSlowNodesParameters(final DatanodeManager datanodeManager,
final String property, final String newVal) throws
ReconfigurationException {
+BlockManager bm = namesystem.getBlockManager();
namesystem.writeLock();
boolean enable;
try {
- if (newVal == null) {
-enable = DFS_NAMENODE_AVOID_SLOW_DATANODE_FOR_READ_DEFAULT;
+ if (property.equals(DFS_NAMENODE_AVOID_SLOW_DATANODE_FOR_READ_KEY)) {
+enable = (newVal == null ?
DFS_NAMENODE_AVOID_SLOW_DATANODE_FOR_READ_DEFAULT :
+Boolean.parseBoolean(newVal));
+datanodeManager.setAvoidSlowDataNodesForReadEnabled(enable);
+ } else if (property.equals(
+DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_KEY))
{
+enable = (newVal == null ?
+
DFS_NAMENODE_BLOCKPLACEMENTPOLICY_EXCLUDE_SLOW_NODES_ENABLED_DEFAULT :
+Boolean.parseBoolean(newVal));
+bm.setExculeSlowDataNodesForWriteEnabled(enable);
} else {
-enable = Boolean.parseBoolean(newVal);
+throw new IllegalArgumentException("Unexpected property " +
+property + "in reconfReplicationParameters");
Review comment:
@tomscut Thank you for your reply.
Fixed and updated PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 681803)
Time Spent: 50m (was: 40m)
> Support to make
> dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
> -
>
> Key: HDFS-16314
> URL: https://issues.apache.org/jira/browse/HDFS-16314
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Consider that make
> dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
> and rapid rollback in case this feature HDFS-16076 unexpected things happen
> in production environment
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16314) Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16314?focusedWorklogId=681805&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681805 ] ASF GitHub Bot logged work on HDFS-16314: - Author: ASF GitHub Bot Created on: 16/Nov/21 03:20 Start Date: 16/Nov/21 03:20 Worklog Time Spent: 10m Work Description: haiyang1987 closed pull request #3651: URL: https://github.com/apache/hadoop/pull/3651 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681805) Time Spent: 1h (was: 50m) > Support to make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > - > > Key: HDFS-16314 > URL: https://issues.apache.org/jira/browse/HDFS-16314 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Consider that make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > and rapid rollback in case this feature HDFS-16076 unexpected things happen > in production environment -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16315) Add metrics related to Transfer and NativeCopy for DataNode
[ https://issues.apache.org/jira/browse/HDFS-16315?focusedWorklogId=681806&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681806 ] ASF GitHub Bot logged work on HDFS-16315: - Author: ASF GitHub Bot Created on: 16/Nov/21 03:21 Start Date: 16/Nov/21 03:21 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3643: URL: https://github.com/apache/hadoop/pull/3643#issuecomment-969782246 @tomscut I'd like to cherry-pick it to lower branches, but there are small conflicts. Could you create another PR for branch-3.3? The variable of `name` in `TestFsDatasetImpl` doesn't exist in branch-3.3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681806) Time Spent: 3h 20m (was: 3h 10m) > Add metrics related to Transfer and NativeCopy for DataNode > --- > > Key: HDFS-16315 > URL: https://issues.apache.org/jira/browse/HDFS-16315 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2021-11-11-08-26-33-074.png > > Time Spent: 3h 20m > Remaining Estimate: 0h > > Datanodes already have Read, Write, Sync and Flush metrics. We should add > NativeCopy and Transfer as well. > Here is a partial look after the change: > !image-2021-11-11-08-26-33-074.png|width=205,height=235! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16315) Add metrics related to Transfer and NativeCopy for DataNode
[ https://issues.apache.org/jira/browse/HDFS-16315?focusedWorklogId=681807&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681807 ] ASF GitHub Bot logged work on HDFS-16315: - Author: ASF GitHub Bot Created on: 16/Nov/21 03:24 Start Date: 16/Nov/21 03:24 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3643: URL: https://github.com/apache/hadoop/pull/3643#issuecomment-969787153 > @tomscut I'd like to cherry-pick it to lower branches, but there are small conflicts. Could you create another PR for branch-3.3? The variable of `name` in `TestFsDatasetImpl` doesn't exist in branch-3.3. Thank you for reminding me. I would love to do this and I will submit a separate PR later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681807) Time Spent: 3.5h (was: 3h 20m) > Add metrics related to Transfer and NativeCopy for DataNode > --- > > Key: HDFS-16315 > URL: https://issues.apache.org/jira/browse/HDFS-16315 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2021-11-11-08-26-33-074.png > > Time Spent: 3.5h > Remaining Estimate: 0h > > Datanodes already have Read, Write, Sync and Flush metrics. We should add > NativeCopy and Transfer as well. > Here is a partial look after the change: > !image-2021-11-11-08-26-33-074.png|width=205,height=235! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16314) Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16314?focusedWorklogId=681808&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681808 ] ASF GitHub Bot logged work on HDFS-16314: - Author: ASF GitHub Bot Created on: 16/Nov/21 03:39 Start Date: 16/Nov/21 03:39 Worklog Time Spent: 10m Work Description: haiyang1987 opened a new pull request #3664: URL: https://github.com/apache/hadoop/pull/3664 ### Description of PR Consider that make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable and rapid rollback in case this feature HDFS-16076 unexpected things happen in production environment Details: HDFS-16314 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681808) Time Spent: 1h 10m (was: 1h) > Support to make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > - > > Key: HDFS-16314 > URL: https://issues.apache.org/jira/browse/HDFS-16314 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Consider that make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > and rapid rollback in case this feature HDFS-16076 unexpected things happen > in production environment -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16314) Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16314?focusedWorklogId=681809&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681809 ] ASF GitHub Bot logged work on HDFS-16314: - Author: ASF GitHub Bot Created on: 16/Nov/21 03:40 Start Date: 16/Nov/21 03:40 Worklog Time Spent: 10m Work Description: haiyang1987 closed pull request #3664: URL: https://github.com/apache/hadoop/pull/3664 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681809) Time Spent: 1h 20m (was: 1h 10m) > Support to make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > - > > Key: HDFS-16314 > URL: https://issues.apache.org/jira/browse/HDFS-16314 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Consider that make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > and rapid rollback in case this feature HDFS-16076 unexpected things happen > in production environment -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16314) Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16314?focusedWorklogId=681813&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681813 ] ASF GitHub Bot logged work on HDFS-16314: - Author: ASF GitHub Bot Created on: 16/Nov/21 03:45 Start Date: 16/Nov/21 03:45 Worklog Time Spent: 10m Work Description: haiyang1987 edited a comment on pull request #3651: URL: https://github.com/apache/hadoop/pull/3651#issuecomment-969825808 Sorry, code branch HDFS-16314 was deleted by mistake, resubmit a PR https://github.com/apache/hadoop/pull/3664 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681813) Time Spent: 1h 50m (was: 1h 40m) > Support to make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > - > > Key: HDFS-16314 > URL: https://issues.apache.org/jira/browse/HDFS-16314 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Consider that make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > and rapid rollback in case this feature HDFS-16076 unexpected things happen > in production environment -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16314) Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16314?focusedWorklogId=681811&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681811 ] ASF GitHub Bot logged work on HDFS-16314: - Author: ASF GitHub Bot Created on: 16/Nov/21 03:45 Start Date: 16/Nov/21 03:45 Worklog Time Spent: 10m Work Description: haiyang1987 commented on pull request #3651: URL: https://github.com/apache/hadoop/pull/3651#issuecomment-969825808 Sorry, code branch HDFS-16314 was deleted by mistake, resubmit a PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681811) Time Spent: 1.5h (was: 1h 20m) > Support to make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > - > > Key: HDFS-16314 > URL: https://issues.apache.org/jira/browse/HDFS-16314 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Consider that make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > and rapid rollback in case this feature HDFS-16076 unexpected things happen > in production environment -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16314) Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16314?focusedWorklogId=681812&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681812 ] ASF GitHub Bot logged work on HDFS-16314: - Author: ASF GitHub Bot Created on: 16/Nov/21 03:45 Start Date: 16/Nov/21 03:45 Worklog Time Spent: 10m Work Description: haiyang1987 opened a new pull request #3664: URL: https://github.com/apache/hadoop/pull/3664 ### Description of PR Consider that make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable and rapid rollback in case this feature HDFS-16076 unexpected things happen in production environment Details: HDFS-16314 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681812) Time Spent: 1h 40m (was: 1.5h) > Support to make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > - > > Key: HDFS-16314 > URL: https://issues.apache.org/jira/browse/HDFS-16314 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > Consider that make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > and rapid rollback in case this feature HDFS-16076 unexpected things happen > in production environment -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16314) Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16314?focusedWorklogId=681814&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681814 ] ASF GitHub Bot logged work on HDFS-16314: - Author: ASF GitHub Bot Created on: 16/Nov/21 03:56 Start Date: 16/Nov/21 03:56 Worklog Time Spent: 10m Work Description: haiyang1987 commented on pull request #3664: URL: https://github.com/apache/hadoop/pull/3664#issuecomment-969834330 @tomscut Thank you for your reply. sorry, the previous PR exception was closed and created a new PR. As you suggested fixed and updated PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681814) Time Spent: 2h (was: 1h 50m) > Support to make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > - > > Key: HDFS-16314 > URL: https://issues.apache.org/jira/browse/HDFS-16314 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > Consider that make > dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable > and rapid rollback in case this feature HDFS-16076 unexpected things happen > in production environment -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16322) The NameNode implementation of ClientProtocol.truncate(...) can cause data loss.
[ https://issues.apache.org/jira/browse/HDFS-16322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17444281#comment-17444281 ] nhaorand commented on HDFS-16322: - Thanks [~shv] for your response. Do you mean that this issue should be addressed by the client? That means we should change ClientProtocol.truncate to @AtMostOnce. > The NameNode implementation of ClientProtocol.truncate(...) can cause data > loss. > > > Key: HDFS-16322 > URL: https://issues.apache.org/jira/browse/HDFS-16322 > Project: Hadoop HDFS > Issue Type: Bug > Environment: The runtime environment is Ubuntu 18.04, Java 1.8.0_222 > and Apache Maven 3.6.0. > The bug can be reproduced by the the testMultipleTruncate() in the > attachment. First, replace the file TestFileTruncate.java under the directory > "hadoop-3.3.1-src/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/" > with the attachment. Then run "mvn test > -Dtest=org.apache.hadoop.hdfs.server.namenode.TestFileTruncate#testMultipleTruncate" > to run the testcase. Finally the "assertFileLength(p, n+newLength)" at 199 > line of TestFileTruncate.java will abort. Because the retry of truncate() > changes the file size and cause data loss. >Reporter: nhaorand >Priority: Major > Attachments: TestFileTruncate.java > > > The NameNode implementation of ClientProtocol.truncate(...) can cause data > loss. If dfsclient drops the first response of a truncate RPC call, the retry > by retry cache will truncate the file again and cause data loss. > HDFS-7926 avoids repeated execution of truncate(...) by checking if the file > is already being truncated with the same length. However, under concurrency, > after the first execution of truncate(...), concurrent requests from other > clients may append new data and change the file length. When truncate(...) is > retried after that, it will find the file has not been truncated with the > same length and truncate it again, which causes data loss. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feilong He updated HDFS-15714: -- Attachment: HDFS-15714-02.patch > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, HDFS-15714-02.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 2.5h > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16315) Add metrics related to Transfer and NativeCopy for DataNode
[ https://issues.apache.org/jira/browse/HDFS-16315?focusedWorklogId=681838&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681838 ] ASF GitHub Bot logged work on HDFS-16315: - Author: ASF GitHub Bot Created on: 16/Nov/21 06:32 Start Date: 16/Nov/21 06:32 Worklog Time Spent: 10m Work Description: tomscut opened a new pull request #3666: URL: https://github.com/apache/hadoop/pull/3666 Backport [HDFS-16315](https://issues.apache.org/jira/browse/HDFS-16315) for branch-3.3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681838) Time Spent: 3h 40m (was: 3.5h) > Add metrics related to Transfer and NativeCopy for DataNode > --- > > Key: HDFS-16315 > URL: https://issues.apache.org/jira/browse/HDFS-16315 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2021-11-11-08-26-33-074.png > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Datanodes already have Read, Write, Sync and Flush metrics. We should add > NativeCopy and Transfer as well. > Here is a partial look after the change: > !image-2021-11-11-08-26-33-074.png|width=205,height=235! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16315) Add metrics related to Transfer and NativeCopy for DataNode
[ https://issues.apache.org/jira/browse/HDFS-16315?focusedWorklogId=681840&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681840 ] ASF GitHub Bot logged work on HDFS-16315: - Author: ASF GitHub Bot Created on: 16/Nov/21 06:37 Start Date: 16/Nov/21 06:37 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3643: URL: https://github.com/apache/hadoop/pull/3643#issuecomment-969915079 > @tomscut I'd like to cherry-pick it to lower branches, but there are small conflicts. Could you create another PR for branch-3.3? The variable of `name` in `TestFsDatasetImpl` doesn't exist in branch-3.3. Hi @tasanuma , I submitted a PR [#3666]( https://github.com/apache/hadoop/pull/3666) for branch-3.3. Please help review it after the build. Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681840) Time Spent: 3h 50m (was: 3h 40m) > Add metrics related to Transfer and NativeCopy for DataNode > --- > > Key: HDFS-16315 > URL: https://issues.apache.org/jira/browse/HDFS-16315 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2021-11-11-08-26-33-074.png > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Datanodes already have Read, Write, Sync and Flush metrics. We should add > NativeCopy and Transfer as well. > Here is a partial look after the change: > !image-2021-11-11-08-26-33-074.png|width=205,height=235! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15714) HDFS Provided Storage Read/Write Mount Support On-the-fly
[ https://issues.apache.org/jira/browse/HDFS-15714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17444290#comment-17444290 ] Feilong He commented on HDFS-15714: --- Uploaded [^HDFS-15714-02.patch] with two commits introduced to fix the following issues: 1) Exclude provided storage in setting up pipeline for append operation. 2) Fix sync failure for truncated data with provided replica. > HDFS Provided Storage Read/Write Mount Support On-the-fly > - > > Key: HDFS-15714 > URL: https://issues.apache.org/jira/browse/HDFS-15714 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Labels: pull-request-available > Attachments: HDFS-15714-01.patch, HDFS-15714-02.patch, > HDFS_Provided_Storage_Design-V1.pdf, HDFS_Provided_Storage_Performance-V1.pdf > > Time Spent: 2.5h > Remaining Estimate: 0h > > HDFS Provided Storage (PS) is a feature to tier HDFS over other file systems. > In HDFS-9806, PROVIDED storage type was introduced to HDFS. Through > configuring external storage with PROVIDED tag for DataNode, user can enable > application to access data stored externally from HDFS side. However, there > are two issues need to be addressed. Firstly, mounting external storage > on-the-fly, namely dynamic mount, is lacking. It is necessary to get it > supported to flexibly combine HDFS with an external storage at runtime. > Secondly, PS write is not supported by current HDFS. But in real > applications, it is common to transfer data bi-directionally for read/write > between HDFS and external storage. > Through this JIRA, we are presenting our work for PS write support and > dynamic mount support for both read & write. Please note in the community > several JIRAs have been filed for these topics. Our work is based on these > previous community work, with new design & implementation to support called > writeBack mount and enable admin to add any mount on-the-fly. We appreciate > those folks in the community for their great contribution! See their pending > JIRAs: HDFS-14805 & HDFS-12090. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14480) Shut down DataNode gracefully when responding to stop-dfs.sh/stop-dfs.cmd
[ https://issues.apache.org/jira/browse/HDFS-14480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feilong He resolved HDFS-14480. --- Resolution: Won't Fix > Shut down DataNode gracefully when responding to stop-dfs.sh/stop-dfs.cmd > - > > Key: HDFS-14480 > URL: https://issues.apache.org/jira/browse/HDFS-14480 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > > Currently, DataNode has #shutdown method to tackle something before shutdown. > But its shutdown hook does't call this method. In HDFS-14401 for HDFS > persistent memory cache optimization, we added clean cache logic in DN's > #shutdown method. And we expect DN will clean up cache during shut down by > stop-dfs.sh/stop-dfs.cmd, which depends on this Jira's patch. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16014) Issue in checking native pmdk lib by 'hadoop checknative' command
[ https://issues.apache.org/jira/browse/HDFS-16014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17444297#comment-17444297 ] Feilong He commented on HDFS-16014: --- [~rakeshr], do you have any comment on this patch? > Issue in checking native pmdk lib by 'hadoop checknative' command > - > > Key: HDFS-16014 > URL: https://issues.apache.org/jira/browse/HDFS-16014 > Project: Hadoop HDFS > Issue Type: Bug > Components: native >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Major > Attachments: HDFS-16014-01.patch > > > In HDFS-14818, we proposed a patch to support checking native pmdk lib. The > expected target is to display hint to user regarding pmdk lib loaded state. > Recently, it was found that pmdk lib was not successfully loaded actually but > the `hadoop checknative` command still tells user that it was. This issue can > be reproduced by moving libpmem.so* from specified installed path to other > place, or directly deleting these libs, after the project is built. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15788) Correct the statement for pmem cache to reflect cache persistence support
[ https://issues.apache.org/jira/browse/HDFS-15788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17444301#comment-17444301 ] Feilong He commented on HDFS-15788: --- [~rakeshr], this patch is just to update document to align with the implementation. If you have any comment, please let me know. > Correct the statement for pmem cache to reflect cache persistence support > - > > Key: HDFS-15788 > URL: https://issues.apache.org/jira/browse/HDFS-15788 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation >Affects Versions: 3.4.0 >Reporter: Feilong He >Assignee: Feilong He >Priority: Minor > Attachments: HDFS-15788-01.patch, HDFS-15788-02.patch > > > Correct the statement for pmem cache to reflect cache persistence support. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16315) Add metrics related to Transfer and NativeCopy for DataNode
[ https://issues.apache.org/jira/browse/HDFS-16315?focusedWorklogId=681844&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681844 ] ASF GitHub Bot logged work on HDFS-16315: - Author: ASF GitHub Bot Created on: 16/Nov/21 07:27 Start Date: 16/Nov/21 07:27 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3666: URL: https://github.com/apache/hadoop/pull/3666#issuecomment-969957949 @tomscut Thanks for creating the PR. +1, pending Jenkins. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681844) Time Spent: 4h (was: 3h 50m) > Add metrics related to Transfer and NativeCopy for DataNode > --- > > Key: HDFS-16315 > URL: https://issues.apache.org/jira/browse/HDFS-16315 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2021-11-11-08-26-33-074.png > > Time Spent: 4h > Remaining Estimate: 0h > > Datanodes already have Read, Write, Sync and Flush metrics. We should add > NativeCopy and Transfer as well. > Here is a partial look after the change: > !image-2021-11-11-08-26-33-074.png|width=205,height=235! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16323) DatanodeHttpServer doesn't require handler state map while retrieving filter handlers
[ https://issues.apache.org/jira/browse/HDFS-16323?focusedWorklogId=681846&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-681846 ] ASF GitHub Bot logged work on HDFS-16323: - Author: ASF GitHub Bot Created on: 16/Nov/21 07:41 Start Date: 16/Nov/21 07:41 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3659: URL: https://github.com/apache/hadoop/pull/3659#issuecomment-969965471 @anuengineer @aajisaka @tasanuma Could you please take a look? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 681846) Time Spent: 40m (was: 0.5h) > DatanodeHttpServer doesn't require handler state map while retrieving filter > handlers > - > > Key: HDFS-16323 > URL: https://issues.apache.org/jira/browse/HDFS-16323 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > DatanodeHttpServer#getFilterHandlers use handler state map just to query if > the given datanode httpserver filter handler class exists in the map and if > not, initialize the Channel handler by invoking specific parameterized > constructor of the class. However, this handler state map is never used to > upsert any data. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
