[jira] [Work logged] (HDFS-16477) [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16477?focusedWorklogId=749853=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749853
 ]

ASF GitHub Bot logged work on HDFS-16477:
-

Author: ASF GitHub Bot
Created on: 30/Mar/22 05:55
Start Date: 30/Mar/22 05:55
Worklog Time Spent: 10m 
  Work Description: ayushtkn commented on a change in pull request #4009:
URL: https://github.com/apache/hadoop/pull/4009#discussion_r838142566



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
##
@@ -482,6 +492,8 @@ private void doTestWhenStoragePolicySetToCOLD() throws 
Exception {
 
 hdfsCluster.triggerHeartbeats();
 dfs.satisfyStoragePolicy(new Path(FILE));
+// Assert metrics.
+assertEquals(1, hdfsCluster.getNamesystem().getPendingSPSPaths());

Review comment:
   it is still there, you added a new test as well to get rid of this only 
right?

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
##
@@ -432,7 +443,6 @@ public void testTraverseWhenParentDeleted() throws 
Exception {
   public void testTraverseWhenRootParentDeleted() throws Exception {
   }
 
-

Review comment:
   nit:
   avoid this




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 749853)
Time Spent: 4h 40m  (was: 4.5h)

> [SPS]: Add metric PendingSPSPaths for getting the number of paths to be 
> processed by SPS
> 
>
> Key: HDFS-16477
> URL: https://issues.apache.org/jira/browse/HDFS-16477
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> Currently we have no idea how many paths are waiting to be processed when 
> using the SPS feature. We should add metric PendingSPSPaths for getting the 
> number of paths to be processed by SPS in NameNode.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDFS-16413) Reconfig dfs usage parameters for datanode

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16413?focusedWorklogId=749804=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749804
 ]

ASF GitHub Bot logged work on HDFS-16413:
-

Author: ASF GitHub Bot
Created on: 30/Mar/22 02:14
Start Date: 30/Mar/22 02:14
Worklog Time Spent: 10m 
  Work Description: tasanuma commented on pull request #3863:
URL: https://github.com/apache/hadoop/pull/3863#issuecomment-1082545102


   +1, pending CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 749804)
Time Spent: 1h 50m  (was: 1h 40m)

> Reconfig dfs usage parameters for datanode
> --
>
> Key: HDFS-16413
> URL: https://issues.apache.org/jira/browse/HDFS-16413
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Reconfig dfs usage parameters for datanode.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDFS-16513) [SBN read] Observer Namenode should not trigger the edits rolling of active Namenode

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16513?focusedWorklogId=749775=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749775
 ]

ASF GitHub Bot logged work on HDFS-16513:
-

Author: ASF GitHub Bot
Created on: 30/Mar/22 01:07
Start Date: 30/Mar/22 01:07
Worklog Time Spent: 10m 
  Work Description: tomscut commented on pull request #4087:
URL: https://github.com/apache/hadoop/pull/4087#issuecomment-1082513955


   To avoid frequent edtis rolling, we should disable OBN from triggering the 
edits rolling of active Namenode. 
   
   Hi @sunchao, please have a look at this. Thank you very much. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 749775)
Time Spent: 1h  (was: 50m)

> [SBN read] Observer Namenode should not trigger the edits rolling of active 
> Namenode
> 
>
> Key: HDFS-16513
> URL: https://issues.apache.org/jira/browse/HDFS-16513
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> To avoid frequent edtis rolling, we should disable OBN from triggering the 
> edits rolling of active Namenode. 
> It is sufficient to retain only the triggering of SNN and the auto rolling of 
> ANN. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDFS-16507) [SBN read] Avoid purging edit log which is in progress

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16507?focusedWorklogId=749772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749772
 ]

ASF GitHub Bot logged work on HDFS-16507:
-

Author: ASF GitHub Bot
Created on: 30/Mar/22 01:03
Start Date: 30/Mar/22 01:03
Worklog Time Spent: 10m 
  Work Description: tomscut edited a comment on pull request #4082:
URL: https://github.com/apache/hadoop/pull/4082#issuecomment-1082509136


   In order not to affect the basic logic, I introduced the suggestion 
@virajjasani mentioned. Just change the `assert` to the 
`Preconditions.checkArgument`. This avoids the potential security implications 
of disabling assertions in production.
   
   @tasanuma @Hexiaoqiao @ayushtkn Please also take a look at this. Thanks a 
lot.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 749772)
Time Spent: 4h 20m  (was: 4h 10m)

> [SBN read] Avoid purging edit log which is in progress
> --
>
> Key: HDFS-16507
> URL: https://issues.apache.org/jira/browse/HDFS-16507
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: tomscut
>Assignee: tomscut
>Priority: Critical
>  Labels: pull-request-available
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> We introduced [Standby Read] feature in branch-3.1.0, but found a FATAL 
> exception. It looks like it's purging edit logs which is in process.
> According to the analysis, I suspect that the editlog which is in progress to 
> be purged(after SNN checkpoint) does not finalize(See HDFS-14317) before ANN 
> rolls edit its self. 
> The stack:
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
>     org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
>     
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
>     
> org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
>     
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
>     
> org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
>     
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
> org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
>     
> org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
>     
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
>     
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
>     java.security.AccessController.doPrivileged(Native Method)
>     javax.security.auth.Subject.doAs(Subject.java:422)
>     
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>     
> org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
>     javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
>     javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>     org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>     
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>     
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
>     
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>     org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>     
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>     org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>     
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>     
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>     
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>     
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>     org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>     
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>     
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>     
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>     
> 

[jira] [Work logged] (HDFS-16507) [SBN read] Avoid purging edit log which is in progress

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16507?focusedWorklogId=749767=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749767
 ]

ASF GitHub Bot logged work on HDFS-16507:
-

Author: ASF GitHub Bot
Created on: 30/Mar/22 00:56
Start Date: 30/Mar/22 00:56
Worklog Time Spent: 10m 
  Work Description: tomscut commented on pull request #4082:
URL: https://github.com/apache/hadoop/pull/4082#issuecomment-1082509136


   In order not to affect the basic logic, just change the `assert` to the 
`Preconditions.checkArgument`. This avoids the potential security implications 
of disabling assertions in production.
   
   @tasanuma @Hexiaoqiao @ayushtkn Please also take a look at this. Thanks a 
lot.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 749767)
Time Spent: 4h 10m  (was: 4h)

> [SBN read] Avoid purging edit log which is in progress
> --
>
> Key: HDFS-16507
> URL: https://issues.apache.org/jira/browse/HDFS-16507
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: tomscut
>Assignee: tomscut
>Priority: Critical
>  Labels: pull-request-available
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> We introduced [Standby Read] feature in branch-3.1.0, but found a FATAL 
> exception. It looks like it's purging edit logs which is in process.
> According to the analysis, I suspect that the editlog which is in progress to 
> be purged(after SNN checkpoint) does not finalize(See HDFS-14317) before ANN 
> rolls edit its self. 
> The stack:
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
>     org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
>     
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
>     
> org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
>     
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
>     
> org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
>     
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
> org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
>     
> org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
>     
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
>     
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
>     java.security.AccessController.doPrivileged(Native Method)
>     javax.security.auth.Subject.doAs(Subject.java:422)
>     
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>     
> org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
>     javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
>     javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>     org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>     
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>     
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
>     
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>     org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>     
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
>     org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>     
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>     
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>     
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>     
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
>     org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>     
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>     
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>     
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>     
> 

[jira] [Work logged] (HDFS-16413) Reconfig dfs usage parameters for datanode

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16413?focusedWorklogId=749746=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749746
 ]

ASF GitHub Bot logged work on HDFS-16413:
-

Author: ASF GitHub Bot
Created on: 30/Mar/22 00:27
Start Date: 30/Mar/22 00:27
Worklog Time Spent: 10m 
  Work Description: tomscut commented on pull request #3863:
URL: https://github.com/apache/hadoop/pull/3863#issuecomment-1082494579


   > @tomscut Thanks for working on this, and sorry it has been a long time 
since the PR was created. It looks mostly good to me. I left some minor 
comments.
   
   Thank you @tasanuma very much for your review and suggestion. It makes sense 
to me. I updated the code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 749746)
Time Spent: 1h 40m  (was: 1.5h)

> Reconfig dfs usage parameters for datanode
> --
>
> Key: HDFS-16413
> URL: https://issues.apache.org/jira/browse/HDFS-16413
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Reconfig dfs usage parameters for datanode.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDFS-16457) Make fs.getspaceused.classname reconfigurable

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16457?focusedWorklogId=749485=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749485
 ]

ASF GitHub Bot logged work on HDFS-16457:
-

Author: ASF GitHub Bot
Created on: 29/Mar/22 17:27
Start Date: 29/Mar/22 17:27
Worklog Time Spent: 10m 
  Work Description: tasanuma commented on pull request #4069:
URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1082168068


   Thanks for working on this, @singer-bin. I'd like to review your PR after 
#3863. I hope you will rebase this PR after merging #3863.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 749485)
Time Spent: 2h 20m  (was: 2h 10m)

> Make fs.getspaceused.classname reconfigurable
> -
>
> Key: HDFS-16457
> URL: https://issues.apache.org/jira/browse/HDFS-16457
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: namenode
>Affects Versions: 3.3.0
>Reporter: yanbin.zhang
>Assignee: yanbin.zhang
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Now if we want to switch fs.getspaceused.classname we need to restart the 
> NameNode. It would be convenient if we can switch it at runtime.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDFS-16413) Reconfig dfs usage parameters for datanode

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16413?focusedWorklogId=749483=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749483
 ]

ASF GitHub Bot logged work on HDFS-16413:
-

Author: ASF GitHub Bot
Created on: 29/Mar/22 17:24
Start Date: 29/Mar/22 17:24
Worklog Time Spent: 10m 
  Work Description: tasanuma commented on a change in pull request #3863:
URL: https://github.com/apache/hadoop/pull/3863#discussion_r837704290



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -854,6 +865,43 @@ private String reconfSlowDiskParameters(String property, 
String newVal)
 }
   }
 
+  private String reconfDfsUsageParameters(String property, String newVal)
+  throws ReconfigurationException {
+String result = null;
+try {
+  LOG.info("Reconfiguring {} to {}", property, newVal);
+  if (property.equals(FS_DU_INTERVAL_KEY)) {
+Preconditions.checkNotNull(data, "FsDatasetSpi has not been 
initialized.");
+long interval = (newVal == null ? FS_DU_INTERVAL_DEFAULT :
+Long.parseLong(newVal));
+result = Long.toString(interval);
+List volumeList = data.getVolumeList();
+for (FsVolumeImpl fsVolume : volumeList) {
+  Map blockPoolSlices = 
fsVolume.getBlockPoolSlices();
+  for (Entry entry : 
blockPoolSlices.entrySet()) {
+entry.getValue().updateDfsUsageConfig(interval, null);
+  }
+}
+  } else if (property.equals(FS_GETSPACEUSED_JITTER_KEY)) {
+Preconditions.checkNotNull(data, "FsDatasetSpi has not been 
initialized.");
+long jitter = (newVal == null ? FS_GETSPACEUSED_JITTER_DEFAULT :
+Long.parseLong(newVal));
+result = Long.toString(jitter);
+List volumeList = data.getVolumeList();
+for (FsVolumeImpl fsVolume : volumeList) {
+  Map blockPoolSlices = 
fsVolume.getBlockPoolSlices();
+  for (Entry entry : 
blockPoolSlices.entrySet()) {
+entry.getValue().updateDfsUsageConfig(null, jitter);
+  }

Review comment:
   ```suggestion
 for (BlockPoolSlice bp : blockPoolSlices.values()) {
   bp.updateDfsUsageConfig(null, jitter);
 }
   ```

##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -854,6 +865,43 @@ private String reconfSlowDiskParameters(String property, 
String newVal)
 }
   }
 
+  private String reconfDfsUsageParameters(String property, String newVal)
+  throws ReconfigurationException {
+String result = null;
+try {
+  LOG.info("Reconfiguring {} to {}", property, newVal);
+  if (property.equals(FS_DU_INTERVAL_KEY)) {
+Preconditions.checkNotNull(data, "FsDatasetSpi has not been 
initialized.");
+long interval = (newVal == null ? FS_DU_INTERVAL_DEFAULT :
+Long.parseLong(newVal));
+result = Long.toString(interval);
+List volumeList = data.getVolumeList();
+for (FsVolumeImpl fsVolume : volumeList) {
+  Map blockPoolSlices = 
fsVolume.getBlockPoolSlices();
+  for (Entry entry : 
blockPoolSlices.entrySet()) {
+entry.getValue().updateDfsUsageConfig(interval, null);
+  }

Review comment:
   We can make it simpler.
   ```suggestion
 for (BlockPoolSlice bp : blockPoolSlices.values()) {
   bp.updateDfsUsageConfig(interval, null);
 }
   ```

##
File path: 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java
##
@@ -154,10 +155,19 @@ boolean running() {
   /**
* How long in between runs of the background refresh.
*/
-  long getRefreshInterval() {
+  public long getRefreshInterval() {

Review comment:
   ```suggestion
 @VisibleForTesting
 public long getRefreshInterval() {
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 749483)
Time Spent: 1.5h  (was: 1h 20m)

> Reconfig dfs usage parameters for datanode
> --
>
> Key: HDFS-16413
> URL: https://issues.apache.org/jira/browse/HDFS-16413
> Project: Hadoop HDFS
>  Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Reconfig 

[jira] [Commented] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication

2022-03-29 Thread Takanobu Asanuma (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514170#comment-17514170
 ] 

Takanobu Asanuma commented on HDFS-16456:
-

Thanks for updating the patch, [~caozhiqiang]. I prefer 
[^HDFS-16456.009.patch]. Some comments on [^HDFS-16456.009.patch]:

1. How about adding getNumOfNonEmptyRacks() in NetworkTopology, and using it in 
BlockPlacementPolicyRackFaultTolerant and FSNamesystem?
{code:java}
public int getNumOfNonEmptyRacks() {  
  return numOfRacks - numOfEmptyRacks;  
}
{code}
2. Sorry, after all, adding {{emptyRacks}} field to 
{{BlockPlacementStatusDefault}} becomes complex. So let's leave it as it was 
before.

3. The following WARN log in NetworkTopology should be DEBUG or removed.
{code:java}
LOG.warn("Current numOfEmptyRacks is {}", numOfEmptyRacks);
{code}
4. I couldn't understand what is the purpose of the field of decommissionNodes 
in NetoworkTopology. Could you elaborate on it?

5. Could you fix the new checkstyle issues in NetworkTopology? I don't think we 
need to follow the same format as the existing source code.

6. Could you use GenericTestUtils.waitFor to wait for decommissioning in 
testPlacementWithOnlyOneNodeInRackDecommission()?

> EC: Decommission a rack with only on dn will fail when the rack number is 
> equal with replication
> 
>
> Key: HDFS-16456
> URL: https://issues.apache.org/jira/browse/HDFS-16456
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: ec, namenode
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Priority: Critical
> Attachments: HDFS-16456.001.patch, HDFS-16456.002.patch, 
> HDFS-16456.003.patch, HDFS-16456.004.patch, HDFS-16456.005.patch, 
> HDFS-16456.006.patch, HDFS-16456.007.patch, HDFS-16456.008.patch, 
> HDFS-16456.009.patch
>
>
> In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
>  # Enable EC policy, such as RS-6-3-1024k.
>  # The rack number in this cluster is equal with or less than the replication 
> number(9)
>  # A rack only has one DN, and decommission this DN.
> The root cause is in 
> BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will 
> give a limit parameter maxNodesPerRack for choose targets. In this scenario, 
> the maxNodesPerRack is 1, which means each rack can only be chosen one 
> datanode.
> {code:java}
>   protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
>...
>     // If more replicas than racks, evenly spread the replicas.
>     // This calculation rounds up.
>     int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> return new int[] {numOfReplicas, maxNodesPerRack};
>   } {code}
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> here will be called, where totalNumOfReplicas=9 and  numOfRacks=9  
> When we decommission one dn which is only one node in its rack, the 
> chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder() 
> will throw NotEnoughReplicasException, but the exception will not be caught 
> and fail to fallback to chooseEvenlyFromRemainingRacks() function.
> When decommission, after choose targets, verifyBlockPlacement() function will 
> return the total rack number contains the invalid rack, and 
> BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false 
> and it will also cause decommission fail.
> {code:java}
>   public BlockPlacementStatus verifyBlockPlacement(DatanodeInfo[] locs,
>       int numberOfReplicas) {
>     if (locs == null)
>       locs = DatanodeDescriptor.EMPTY_ARRAY;
>     if (!clusterMap.hasClusterEverBeenMultiRack()) {
>       // only one rack
>       return new BlockPlacementStatusDefault(1, 1, 1);
>     }
>     // Count locations on different racks.
>     Set racks = new HashSet<>();
>     for (DatanodeInfo dn : locs) {
>       racks.add(dn.getNetworkLocation());
>     }
>     return new BlockPlacementStatusDefault(racks.size(), numberOfReplicas,
>         clusterMap.getNumOfRacks());
>   } {code}
> {code:java}
>   public boolean isPlacementPolicySatisfied() {
>     return requiredRacks <= currentRacks || currentRacks >= totalRacks;
>   }{code}
> According to the above description, we should make the below modify to fix it:
>  # In startDecommission() or stopDecommission(), we should also change the 
> numOfRacks in class NetworkTopology. Or choose targets may fail for the 
> maxNodesPerRack is too small. And even choose targets success, 
> isPlacementPolicySatisfied will also return false cause decommission fail.
>  # In BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder(), the first 
> chooseOnce() function should also be put in try..catch..., or it will not 
> fallback to call chooseEvenlyFromRemainingRacks() when 

[jira] [Updated] (HDFS-16523) Fix dependency error in hadoop-hdfs on M1 Mac

2022-03-29 Thread Steve Loughran (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HDFS-16523:
--
Affects Version/s: 3.4.0

> Fix dependency error in hadoop-hdfs on M1 Mac
> -
>
> Key: HDFS-16523
> URL: https://issues.apache.org/jira/browse/HDFS-16523
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.4.0
> Environment: M1 Pro Mac
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> hadoop-hdfs build is failing on docker with M1 Mac.
> {code}
> [WARNING]
> Dependency convergence error for
> org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided paths to
> dependency are:
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
>   +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.openlabtesting.leveldbjni:leveldbjni:jar:1.8:provided
>   +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided
> and
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
>   +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.fusesource.leveldbjni:leveldbjni-osx:jar:1.8:provided
>   +-org.fusesource.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.9:provided
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Resolved] (HDFS-16523) Fix dependency error in hadoop-hdfs on M1 Mac

2022-03-29 Thread Steve Loughran (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved HDFS-16523.
---
Fix Version/s: 3.4.0
   Resolution: Fixed

> Fix dependency error in hadoop-hdfs on M1 Mac
> -
>
> Key: HDFS-16523
> URL: https://issues.apache.org/jira/browse/HDFS-16523
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: build
>Affects Versions: 3.4.0
> Environment: M1 Pro Mac
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> hadoop-hdfs build is failing on docker with M1 Mac.
> {code}
> [WARNING]
> Dependency convergence error for
> org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided paths to
> dependency are:
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
>   +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.openlabtesting.leveldbjni:leveldbjni:jar:1.8:provided
>   +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided
> and
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
>   +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.fusesource.leveldbjni:leveldbjni-osx:jar:1.8:provided
>   +-org.fusesource.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.9:provided
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Commented] (HDFS-16523) Fix dependency error in hadoop-hdfs on M1 Mac

2022-03-29 Thread Steve Loughran (Jira)


[ 
https://issues.apache.org/jira/browse/HDFS-16523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17514159#comment-17514159
 ] 

Steve Loughran commented on HDFS-16523:
---

merged into trunk; backport as needed. 

> Fix dependency error in hadoop-hdfs on M1 Mac
> -
>
> Key: HDFS-16523
> URL: https://issues.apache.org/jira/browse/HDFS-16523
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: build
> Environment: M1 Pro Mac
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> hadoop-hdfs build is failing on docker with M1 Mac.
> {code}
> [WARNING]
> Dependency convergence error for
> org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided paths to
> dependency are:
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
>   +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.openlabtesting.leveldbjni:leveldbjni:jar:1.8:provided
>   +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided
> and
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
>   +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.fusesource.leveldbjni:leveldbjni-osx:jar:1.8:provided
>   +-org.fusesource.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.9:provided
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDFS-16523) Fix dependency error in hadoop-hdfs on M1 Mac

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16523?focusedWorklogId=749410=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749410
 ]

ASF GitHub Bot logged work on HDFS-16523:
-

Author: ASF GitHub Bot
Created on: 29/Mar/22 15:45
Start Date: 29/Mar/22 15:45
Worklog Time Spent: 10m 
  Work Description: steveloughran merged pull request #4112:
URL: https://github.com/apache/hadoop/pull/4112


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 749410)
Time Spent: 0.5h  (was: 20m)

> Fix dependency error in hadoop-hdfs on M1 Mac
> -
>
> Key: HDFS-16523
> URL: https://issues.apache.org/jira/browse/HDFS-16523
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: build
> Environment: M1 Pro Mac
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> hadoop-hdfs build is failing on docker with M1 Mac.
> {code}
> [WARNING]
> Dependency convergence error for
> org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided paths to
> dependency are:
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
>   +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.openlabtesting.leveldbjni:leveldbjni:jar:1.8:provided
>   +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided
> and
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
>   +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.fusesource.leveldbjni:leveldbjni-osx:jar:1.8:provided
>   +-org.fusesource.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.9:provided
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org



[jira] [Work logged] (HDFS-16520) Improve EC pread: avoid potential reading whole block

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16520?focusedWorklogId=749392=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749392
 ]

ASF GitHub Bot logged work on HDFS-16520:
-

Author: ASF GitHub Bot
Created on: 29/Mar/22 15:21
Start Date: 29/Mar/22 15:21
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #4104:
URL: https://github.com/apache/hadoop/pull/4104#issuecomment-1082009485


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 39s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 1 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  12m 59s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  23m 19s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   6m  5s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   5m 37s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  checkstyle  |   1m 12s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   2m 32s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m 49s |  |  trunk passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   2m 16s |  |  trunk passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   5m 45s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  22m 41s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 27s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   2m  7s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   5m 44s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   5m 44s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   5m 32s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  javac  |   5m 32s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   1m  2s | 
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4104/3/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
 |  hadoop-hdfs-project: The patch generated 3 new + 29 unchanged - 0 fixed = 
32 total (was 29)  |
   | +1 :green_heart: |  mvnsite  |   2m 11s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   1m 29s |  |  the patch passed with JDK 
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   2m  0s |  |  the patch passed with JDK 
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07  |
   | +1 :green_heart: |  spotbugs  |   5m 49s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  22m 41s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  |   2m 23s |  |  hadoop-hdfs-client in the patch 
passed.  |
   | +1 :green_heart: |  unit  | 226m  1s |  |  hadoop-hdfs in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   0m 47s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 361m  8s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4104/3/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/4104 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux 259ffc61732c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 3dac78493d53795dbe739c9942e9594b1b84da6a |
   | Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
   |  

[jira] [Work logged] (HDFS-16518) KeyProviderCache close cached KeyProvider with Hadoop ShutdownHookManager

2022-03-29 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/HDFS-16518?focusedWorklogId=749335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749335
 ]

ASF GitHub Bot logged work on HDFS-16518:
-

Author: ASF GitHub Bot
Created on: 29/Mar/22 13:41
Start Date: 29/Mar/22 13:41
Worklog Time Spent: 10m 
  Work Description: aajisaka commented on pull request #4100:
URL: https://github.com/apache/hadoop/pull/4100#issuecomment-1081888202


   Hi @omalley - it is committed to trunk, branch-3.3, and branch-2.10 without 
your approval in either JIRA or GitHub PR. Do you have any reason to approve 
the change?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 749335)
Time Spent: 1h 50m  (was: 1h 40m)

> KeyProviderCache close cached KeyProvider with Hadoop ShutdownHookManager
> -
>
> Key: HDFS-16518
> URL: https://issues.apache.org/jira/browse/HDFS-16518
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Affects Versions: 2.10.0
>Reporter: Lei Yang
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> KeyProvider implements Closable interface but some custom implementation of 
> KeyProvider also needs explicit close in KeyProviderCache. An example is to 
> use custom KeyProvider in DFSClient to integrate read encrypted file on HDFS. 
> KeyProvider  currently gets closed in KeyProviderCache only when cache entry 
> is expired or invalidated. In some cases, this is not happening. This seems 
> related to guava cache.
> This patch is to use hadoop JVM shutdownhookManager to globally cleanup cache 
> entries and thus close KeyProvider using cache hook right after filesystem 
> instance gets closed in a deterministic way.
> {code:java}
> Class KeyProviderCache
> ...
>  public KeyProviderCache(long expiryMs) {
>   cache = CacheBuilder.newBuilder()
> .expireAfterAccess(expiryMs, TimeUnit.MILLISECONDS)
> .removalListener(new RemovalListener() {
>   @Override
>   public void onRemoval(
>   @Nonnull RemovalNotification notification) {
> try {
>   assert notification.getValue() != null;
>   notification.getValue().close();
> } catch (Throwable e) {
>   LOG.error(
>   "Error closing KeyProvider with uri ["
>   + notification.getKey() + "]", e);
> }
>   }
> })
> .build(); 
> }{code}
> We could have made a new function KeyProviderCache#close, have each DFSClient 
> call this function and close KeyProvider at the end of each DFSClient#close 
> call but it will expose another problem to potentially close global cache 
> among different DFSClient instances.
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org