[jira] [Resolved] (HDFS-16498) Fix NPE for checkBlockReportLease
[ https://issues.apache.org/jira/browse/HDFS-16498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-16498. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to trunk. Thanks @tomscut for your contribution! > Fix NPE for checkBlockReportLease > - > > Key: HDFS-16498 > URL: https://issues.apache.org/jira/browse/HDFS-16498 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2022-03-09-20-35-22-028.png, screenshot-1.png > > Time Spent: 5h 20m > Remaining Estimate: 0h > > During the restart of Namenode, a Datanode is not registered, but this > Datanode triggers FBR, which causes NPE. > !image-2022-03-09-20-35-22-028.png|width=871,height=158! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16498) Fix NPE for checkBlockReportLease
[ https://issues.apache.org/jira/browse/HDFS-16498?focusedWorklogId=749895&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749895 ] ASF GitHub Bot logged work on HDFS-16498: - Author: ASF GitHub Bot Created on: 30/Mar/22 06:58 Start Date: 30/Mar/22 06:58 Worklog Time Spent: 10m Work Description: Hexiaoqiao commented on pull request #4057: URL: https://github.com/apache/hadoop/pull/4057#issuecomment-1082699279 Committed to trunk. Thanks @tomscut for your contribution! Thanks @ayushtkn for your reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 749895) Time Spent: 5h 20m (was: 5h 10m) > Fix NPE for checkBlockReportLease > - > > Key: HDFS-16498 > URL: https://issues.apache.org/jira/browse/HDFS-16498 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2022-03-09-20-35-22-028.png, screenshot-1.png > > Time Spent: 5h 20m > Remaining Estimate: 0h > > During the restart of Namenode, a Datanode is not registered, but this > Datanode triggers FBR, which causes NPE. > !image-2022-03-09-20-35-22-028.png|width=871,height=158! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16498) Fix NPE for checkBlockReportLease
[ https://issues.apache.org/jira/browse/HDFS-16498?focusedWorklogId=749894&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749894 ] ASF GitHub Bot logged work on HDFS-16498: - Author: ASF GitHub Bot Created on: 30/Mar/22 06:58 Start Date: 30/Mar/22 06:58 Worklog Time Spent: 10m Work Description: Hexiaoqiao closed pull request #4057: URL: https://github.com/apache/hadoop/pull/4057 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 749894) Time Spent: 5h 10m (was: 5h) > Fix NPE for checkBlockReportLease > - > > Key: HDFS-16498 > URL: https://issues.apache.org/jira/browse/HDFS-16498 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2022-03-09-20-35-22-028.png, screenshot-1.png > > Time Spent: 5h 10m > Remaining Estimate: 0h > > During the restart of Namenode, a Datanode is not registered, but this > Datanode triggers FBR, which causes NPE. > !image-2022-03-09-20-35-22-028.png|width=871,height=158! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16457) Make fs.getspaceused.classname reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16457?focusedWorklogId=749869&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749869 ] ASF GitHub Bot logged work on HDFS-16457: - Author: ASF GitHub Bot Created on: 30/Mar/22 06:25 Start Date: 30/Mar/22 06:25 Worklog Time Spent: 10m Work Description: singer-bin commented on pull request #4069: URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1082673023 @tasanuma Thanks for your comment, I will. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 749869) Time Spent: 2.5h (was: 2h 20m) > Make fs.getspaceused.classname reconfigurable > - > > Key: HDFS-16457 > URL: https://issues.apache.org/jira/browse/HDFS-16457 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: yanbin.zhang >Assignee: yanbin.zhang >Priority: Major > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > Now if we want to switch fs.getspaceused.classname we need to restart the > NameNode. It would be convenient if we can switch it at runtime. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16477) [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS
[ https://issues.apache.org/jira/browse/HDFS-16477?focusedWorklogId=749862&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749862 ] ASF GitHub Bot logged work on HDFS-16477: - Author: ASF GitHub Bot Created on: 30/Mar/22 06:16 Start Date: 30/Mar/22 06:16 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #4009: URL: https://github.com/apache/hadoop/pull/4009#issuecomment-1082664700 > TestRouterDistCpProcedure seems to be failing with timeout. Doesn't look related. If you have bandwidth can track & increase the timeout for it in a separate jira Thank you very much, I will open a new JIRA to do this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 749862) Time Spent: 5h 10m (was: 5h) > [SPS]: Add metric PendingSPSPaths for getting the number of paths to be > processed by SPS > > > Key: HDFS-16477 > URL: https://issues.apache.org/jira/browse/HDFS-16477 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > > Currently we have no idea how many paths are waiting to be processed when > using the SPS feature. We should add metric PendingSPSPaths for getting the > number of paths to be processed by SPS in NameNode. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16477) [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS
[
https://issues.apache.org/jira/browse/HDFS-16477?focusedWorklogId=749859&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749859
]
ASF GitHub Bot logged work on HDFS-16477:
-
Author: ASF GitHub Bot
Created on: 30/Mar/22 06:12
Start Date: 30/Mar/22 06:12
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #4009:
URL: https://github.com/apache/hadoop/pull/4009#discussion_r838152418
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
##
@@ -432,7 +443,6 @@ public void testTraverseWhenParentDeleted() throws
Exception {
public void testTraverseWhenRootParentDeleted() throws Exception {
}
-
Review comment:
OK. Let me roll back it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 749859)
Time Spent: 5h (was: 4h 50m)
> [SPS]: Add metric PendingSPSPaths for getting the number of paths to be
> processed by SPS
>
>
> Key: HDFS-16477
> URL: https://issues.apache.org/jira/browse/HDFS-16477
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Currently we have no idea how many paths are waiting to be processed when
> using the SPS feature. We should add metric PendingSPSPaths for getting the
> number of paths to be processed by SPS in NameNode.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16477) [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS
[
https://issues.apache.org/jira/browse/HDFS-16477?focusedWorklogId=749858&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749858
]
ASF GitHub Bot logged work on HDFS-16477:
-
Author: ASF GitHub Bot
Created on: 30/Mar/22 06:10
Start Date: 30/Mar/22 06:10
Worklog Time Spent: 10m
Work Description: tomscut commented on a change in pull request #4009:
URL: https://github.com/apache/hadoop/pull/4009#discussion_r838151716
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
##
@@ -482,6 +492,8 @@ private void doTestWhenStoragePolicySetToCOLD() throws
Exception {
hdfsCluster.triggerHeartbeats();
dfs.satisfyStoragePolicy(new Path(FILE));
+// Assert metrics.
+assertEquals(1, hdfsCluster.getNamesystem().getPendingSPSPaths());
Review comment:
Thanks for your comments. I'm sorry I left that out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 749858)
Time Spent: 4h 50m (was: 4h 40m)
> [SPS]: Add metric PendingSPSPaths for getting the number of paths to be
> processed by SPS
>
>
> Key: HDFS-16477
> URL: https://issues.apache.org/jira/browse/HDFS-16477
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Currently we have no idea how many paths are waiting to be processed when
> using the SPS feature. We should add metric PendingSPSPaths for getting the
> number of paths to be processed by SPS in NameNode.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16477) [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS
[
https://issues.apache.org/jira/browse/HDFS-16477?focusedWorklogId=749853&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749853
]
ASF GitHub Bot logged work on HDFS-16477:
-
Author: ASF GitHub Bot
Created on: 30/Mar/22 05:55
Start Date: 30/Mar/22 05:55
Worklog Time Spent: 10m
Work Description: ayushtkn commented on a change in pull request #4009:
URL: https://github.com/apache/hadoop/pull/4009#discussion_r838142566
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
##
@@ -482,6 +492,8 @@ private void doTestWhenStoragePolicySetToCOLD() throws
Exception {
hdfsCluster.triggerHeartbeats();
dfs.satisfyStoragePolicy(new Path(FILE));
+// Assert metrics.
+assertEquals(1, hdfsCluster.getNamesystem().getPendingSPSPaths());
Review comment:
it is still there, you added a new test as well to get rid of this only
right?
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/sps/TestExternalStoragePolicySatisfier.java
##
@@ -432,7 +443,6 @@ public void testTraverseWhenParentDeleted() throws
Exception {
public void testTraverseWhenRootParentDeleted() throws Exception {
}
-
Review comment:
nit:
avoid this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 749853)
Time Spent: 4h 40m (was: 4.5h)
> [SPS]: Add metric PendingSPSPaths for getting the number of paths to be
> processed by SPS
>
>
> Key: HDFS-16477
> URL: https://issues.apache.org/jira/browse/HDFS-16477
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Currently we have no idea how many paths are waiting to be processed when
> using the SPS feature. We should add metric PendingSPSPaths for getting the
> number of paths to be processed by SPS in NameNode.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16413) Reconfig dfs usage parameters for datanode
[ https://issues.apache.org/jira/browse/HDFS-16413?focusedWorklogId=749804&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749804 ] ASF GitHub Bot logged work on HDFS-16413: - Author: ASF GitHub Bot Created on: 30/Mar/22 02:14 Start Date: 30/Mar/22 02:14 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #3863: URL: https://github.com/apache/hadoop/pull/3863#issuecomment-1082545102 +1, pending CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 749804) Time Spent: 1h 50m (was: 1h 40m) > Reconfig dfs usage parameters for datanode > -- > > Key: HDFS-16413 > URL: https://issues.apache.org/jira/browse/HDFS-16413 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > Reconfig dfs usage parameters for datanode. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16513) [SBN read] Observer Namenode should not trigger the edits rolling of active Namenode
[ https://issues.apache.org/jira/browse/HDFS-16513?focusedWorklogId=749775&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749775 ] ASF GitHub Bot logged work on HDFS-16513: - Author: ASF GitHub Bot Created on: 30/Mar/22 01:07 Start Date: 30/Mar/22 01:07 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #4087: URL: https://github.com/apache/hadoop/pull/4087#issuecomment-1082513955 To avoid frequent edtis rolling, we should disable OBN from triggering the edits rolling of active Namenode. Hi @sunchao, please have a look at this. Thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 749775) Time Spent: 1h (was: 50m) > [SBN read] Observer Namenode should not trigger the edits rolling of active > Namenode > > > Key: HDFS-16513 > URL: https://issues.apache.org/jira/browse/HDFS-16513 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > To avoid frequent edtis rolling, we should disable OBN from triggering the > edits rolling of active Namenode. > It is sufficient to retain only the triggering of SNN and the auto rolling of > ANN. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16507) [SBN read] Avoid purging edit log which is in progress
[
https://issues.apache.org/jira/browse/HDFS-16507?focusedWorklogId=749772&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749772
]
ASF GitHub Bot logged work on HDFS-16507:
-
Author: ASF GitHub Bot
Created on: 30/Mar/22 01:03
Start Date: 30/Mar/22 01:03
Worklog Time Spent: 10m
Work Description: tomscut edited a comment on pull request #4082:
URL: https://github.com/apache/hadoop/pull/4082#issuecomment-1082509136
In order not to affect the basic logic, I introduced the suggestion
@virajjasani mentioned. Just change the `assert` to the
`Preconditions.checkArgument`. This avoids the potential security implications
of disabling assertions in production.
@tasanuma @Hexiaoqiao @ayushtkn Please also take a look at this. Thanks a
lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 749772)
Time Spent: 4h 20m (was: 4h 10m)
> [SBN read] Avoid purging edit log which is in progress
> --
>
> Key: HDFS-16507
> URL: https://issues.apache.org/jira/browse/HDFS-16507
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: tomscut
>Assignee: tomscut
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> We introduced [Standby Read] feature in branch-3.1.0, but found a FATAL
> exception. It looks like it's purging edit logs which is in process.
> According to the analysis, I suspect that the editlog which is in progress to
> be purged(after SNN checkpoint) does not finalize(See HDFS-14317) before ANN
> rolls edit its self.
> The stack:
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
>
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
>
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
> org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>
> org.
[jira] [Work logged] (HDFS-16507) [SBN read] Avoid purging edit log which is in progress
[
https://issues.apache.org/jira/browse/HDFS-16507?focusedWorklogId=749767&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749767
]
ASF GitHub Bot logged work on HDFS-16507:
-
Author: ASF GitHub Bot
Created on: 30/Mar/22 00:56
Start Date: 30/Mar/22 00:56
Worklog Time Spent: 10m
Work Description: tomscut commented on pull request #4082:
URL: https://github.com/apache/hadoop/pull/4082#issuecomment-1082509136
In order not to affect the basic logic, just change the `assert` to the
`Preconditions.checkArgument`. This avoids the potential security implications
of disabling assertions in production.
@tasanuma @Hexiaoqiao @ayushtkn Please also take a look at this. Thanks a
lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 749767)
Time Spent: 4h 10m (was: 4h)
> [SBN read] Avoid purging edit log which is in progress
> --
>
> Key: HDFS-16507
> URL: https://issues.apache.org/jira/browse/HDFS-16507
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: tomscut
>Assignee: tomscut
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> We introduced [Standby Read] feature in branch-3.1.0, but found a FATAL
> exception. It looks like it's purging edit logs which is in process.
> According to the analysis, I suspect that the editlog which is in progress to
> be purged(after SNN checkpoint) does not finalize(See HDFS-14317) before ANN
> rolls edit its self.
> The stack:
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
>
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
>
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
> org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCol
[jira] [Work logged] (HDFS-16413) Reconfig dfs usage parameters for datanode
[ https://issues.apache.org/jira/browse/HDFS-16413?focusedWorklogId=749746&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749746 ] ASF GitHub Bot logged work on HDFS-16413: - Author: ASF GitHub Bot Created on: 30/Mar/22 00:27 Start Date: 30/Mar/22 00:27 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3863: URL: https://github.com/apache/hadoop/pull/3863#issuecomment-1082494579 > @tomscut Thanks for working on this, and sorry it has been a long time since the PR was created. It looks mostly good to me. I left some minor comments. Thank you @tasanuma very much for your review and suggestion. It makes sense to me. I updated the code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 749746) Time Spent: 1h 40m (was: 1.5h) > Reconfig dfs usage parameters for datanode > -- > > Key: HDFS-16413 > URL: https://issues.apache.org/jira/browse/HDFS-16413 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > Reconfig dfs usage parameters for datanode. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16457) Make fs.getspaceused.classname reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16457?focusedWorklogId=749485&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749485 ] ASF GitHub Bot logged work on HDFS-16457: - Author: ASF GitHub Bot Created on: 29/Mar/22 17:27 Start Date: 29/Mar/22 17:27 Worklog Time Spent: 10m Work Description: tasanuma commented on pull request #4069: URL: https://github.com/apache/hadoop/pull/4069#issuecomment-1082168068 Thanks for working on this, @singer-bin. I'd like to review your PR after #3863. I hope you will rebase this PR after merging #3863. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 749485) Time Spent: 2h 20m (was: 2h 10m) > Make fs.getspaceused.classname reconfigurable > - > > Key: HDFS-16457 > URL: https://issues.apache.org/jira/browse/HDFS-16457 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.0 >Reporter: yanbin.zhang >Assignee: yanbin.zhang >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Now if we want to switch fs.getspaceused.classname we need to restart the > NameNode. It would be convenient if we can switch it at runtime. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16413) Reconfig dfs usage parameters for datanode
[
https://issues.apache.org/jira/browse/HDFS-16413?focusedWorklogId=749483&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749483
]
ASF GitHub Bot logged work on HDFS-16413:
-
Author: ASF GitHub Bot
Created on: 29/Mar/22 17:24
Start Date: 29/Mar/22 17:24
Worklog Time Spent: 10m
Work Description: tasanuma commented on a change in pull request #3863:
URL: https://github.com/apache/hadoop/pull/3863#discussion_r837704290
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -854,6 +865,43 @@ private String reconfSlowDiskParameters(String property,
String newVal)
}
}
+ private String reconfDfsUsageParameters(String property, String newVal)
+ throws ReconfigurationException {
+String result = null;
+try {
+ LOG.info("Reconfiguring {} to {}", property, newVal);
+ if (property.equals(FS_DU_INTERVAL_KEY)) {
+Preconditions.checkNotNull(data, "FsDatasetSpi has not been
initialized.");
+long interval = (newVal == null ? FS_DU_INTERVAL_DEFAULT :
+Long.parseLong(newVal));
+result = Long.toString(interval);
+List volumeList = data.getVolumeList();
+for (FsVolumeImpl fsVolume : volumeList) {
+ Map blockPoolSlices =
fsVolume.getBlockPoolSlices();
+ for (Entry entry :
blockPoolSlices.entrySet()) {
+entry.getValue().updateDfsUsageConfig(interval, null);
+ }
+}
+ } else if (property.equals(FS_GETSPACEUSED_JITTER_KEY)) {
+Preconditions.checkNotNull(data, "FsDatasetSpi has not been
initialized.");
+long jitter = (newVal == null ? FS_GETSPACEUSED_JITTER_DEFAULT :
+Long.parseLong(newVal));
+result = Long.toString(jitter);
+List volumeList = data.getVolumeList();
+for (FsVolumeImpl fsVolume : volumeList) {
+ Map blockPoolSlices =
fsVolume.getBlockPoolSlices();
+ for (Entry entry :
blockPoolSlices.entrySet()) {
+entry.getValue().updateDfsUsageConfig(null, jitter);
+ }
Review comment:
```suggestion
for (BlockPoolSlice bp : blockPoolSlices.values()) {
bp.updateDfsUsageConfig(null, jitter);
}
```
##
File path:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
##
@@ -854,6 +865,43 @@ private String reconfSlowDiskParameters(String property,
String newVal)
}
}
+ private String reconfDfsUsageParameters(String property, String newVal)
+ throws ReconfigurationException {
+String result = null;
+try {
+ LOG.info("Reconfiguring {} to {}", property, newVal);
+ if (property.equals(FS_DU_INTERVAL_KEY)) {
+Preconditions.checkNotNull(data, "FsDatasetSpi has not been
initialized.");
+long interval = (newVal == null ? FS_DU_INTERVAL_DEFAULT :
+Long.parseLong(newVal));
+result = Long.toString(interval);
+List volumeList = data.getVolumeList();
+for (FsVolumeImpl fsVolume : volumeList) {
+ Map blockPoolSlices =
fsVolume.getBlockPoolSlices();
+ for (Entry entry :
blockPoolSlices.entrySet()) {
+entry.getValue().updateDfsUsageConfig(interval, null);
+ }
Review comment:
We can make it simpler.
```suggestion
for (BlockPoolSlice bp : blockPoolSlices.values()) {
bp.updateDfsUsageConfig(interval, null);
}
```
##
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CachingGetSpaceUsed.java
##
@@ -154,10 +155,19 @@ boolean running() {
/**
* How long in between runs of the background refresh.
*/
- long getRefreshInterval() {
+ public long getRefreshInterval() {
Review comment:
```suggestion
@VisibleForTesting
public long getRefreshInterval() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 749483)
Time Spent: 1.5h (was: 1h 20m)
> Reconfig dfs usage parameters for datanode
> --
>
> Key: HDFS-16413
> URL: https://issues.apache.org/jira/browse/HDFS-16413
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Recon
[jira] [Commented] (HDFS-16456) EC: Decommission a rack with only on dn will fail when the rack number is equal with replication
[
https://issues.apache.org/jira/browse/HDFS-16456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514170#comment-17514170
]
Takanobu Asanuma commented on HDFS-16456:
-
Thanks for updating the patch, [~caozhiqiang]. I prefer
[^HDFS-16456.009.patch]. Some comments on [^HDFS-16456.009.patch]:
1. How about adding getNumOfNonEmptyRacks() in NetworkTopology, and using it in
BlockPlacementPolicyRackFaultTolerant and FSNamesystem?
{code:java}
public int getNumOfNonEmptyRacks() {
return numOfRacks - numOfEmptyRacks;
}
{code}
2. Sorry, after all, adding {{emptyRacks}} field to
{{BlockPlacementStatusDefault}} becomes complex. So let's leave it as it was
before.
3. The following WARN log in NetworkTopology should be DEBUG or removed.
{code:java}
LOG.warn("Current numOfEmptyRacks is {}", numOfEmptyRacks);
{code}
4. I couldn't understand what is the purpose of the field of decommissionNodes
in NetoworkTopology. Could you elaborate on it?
5. Could you fix the new checkstyle issues in NetworkTopology? I don't think we
need to follow the same format as the existing source code.
6. Could you use GenericTestUtils.waitFor to wait for decommissioning in
testPlacementWithOnlyOneNodeInRackDecommission()?
> EC: Decommission a rack with only on dn will fail when the rack number is
> equal with replication
>
>
> Key: HDFS-16456
> URL: https://issues.apache.org/jira/browse/HDFS-16456
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, namenode
>Affects Versions: 3.4.0
>Reporter: caozhiqiang
>Priority: Critical
> Attachments: HDFS-16456.001.patch, HDFS-16456.002.patch,
> HDFS-16456.003.patch, HDFS-16456.004.patch, HDFS-16456.005.patch,
> HDFS-16456.006.patch, HDFS-16456.007.patch, HDFS-16456.008.patch,
> HDFS-16456.009.patch
>
>
> In below scenario, decommission will fail by TOO_MANY_NODES_ON_RACK reason:
> # Enable EC policy, such as RS-6-3-1024k.
> # The rack number in this cluster is equal with or less than the replication
> number(9)
> # A rack only has one DN, and decommission this DN.
> The root cause is in
> BlockPlacementPolicyRackFaultTolerant::getMaxNodesPerRack() function, it will
> give a limit parameter maxNodesPerRack for choose targets. In this scenario,
> the maxNodesPerRack is 1, which means each rack can only be chosen one
> datanode.
> {code:java}
> protected int[] getMaxNodesPerRack(int numOfChosen, int numOfReplicas) {
>...
> // If more replicas than racks, evenly spread the replicas.
> // This calculation rounds up.
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> return new int[] {numOfReplicas, maxNodesPerRack};
> } {code}
> int maxNodesPerRack = (totalNumOfReplicas - 1) / numOfRacks + 1;
> here will be called, where totalNumOfReplicas=9 and numOfRacks=9
> When we decommission one dn which is only one node in its rack, the
> chooseOnce() in BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder()
> will throw NotEnoughReplicasException, but the exception will not be caught
> and fail to fallback to chooseEvenlyFromRemainingRacks() function.
> When decommission, after choose targets, verifyBlockPlacement() function will
> return the total rack number contains the invalid rack, and
> BlockPlacementStatusDefault::isPlacementPolicySatisfied() will return false
> and it will also cause decommission fail.
> {code:java}
> public BlockPlacementStatus verifyBlockPlacement(DatanodeInfo[] locs,
> int numberOfReplicas) {
> if (locs == null)
> locs = DatanodeDescriptor.EMPTY_ARRAY;
> if (!clusterMap.hasClusterEverBeenMultiRack()) {
> // only one rack
> return new BlockPlacementStatusDefault(1, 1, 1);
> }
> // Count locations on different racks.
> Set racks = new HashSet<>();
> for (DatanodeInfo dn : locs) {
> racks.add(dn.getNetworkLocation());
> }
> return new BlockPlacementStatusDefault(racks.size(), numberOfReplicas,
> clusterMap.getNumOfRacks());
> } {code}
> {code:java}
> public boolean isPlacementPolicySatisfied() {
> return requiredRacks <= currentRacks || currentRacks >= totalRacks;
> }{code}
> According to the above description, we should make the below modify to fix it:
> # In startDecommission() or stopDecommission(), we should also change the
> numOfRacks in class NetworkTopology. Or choose targets may fail for the
> maxNodesPerRack is too small. And even choose targets success,
> isPlacementPolicySatisfied will also return false cause decommission fail.
> # In BlockPlacementPolicyRackFaultTolerant::chooseTargetInOrder(), the first
> chooseOnce() function should also be put in try..catch..., or it will not
> fallback to call chooseEvenlyFromRemainin
[jira] [Updated] (HDFS-16523) Fix dependency error in hadoop-hdfs on M1 Mac
[
https://issues.apache.org/jira/browse/HDFS-16523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16523:
--
Affects Version/s: 3.4.0
> Fix dependency error in hadoop-hdfs on M1 Mac
> -
>
> Key: HDFS-16523
> URL: https://issues.apache.org/jira/browse/HDFS-16523
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
> Environment: M1 Pro Mac
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> hadoop-hdfs build is failing on docker with M1 Mac.
> {code}
> [WARNING]
> Dependency convergence error for
> org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided paths to
> dependency are:
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
> +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.openlabtesting.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided
> and
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
> +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.fusesource.leveldbjni:leveldbjni-osx:jar:1.8:provided
> +-org.fusesource.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.9:provided
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16523) Fix dependency error in hadoop-hdfs on M1 Mac
[
https://issues.apache.org/jira/browse/HDFS-16523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HDFS-16523.
---
Fix Version/s: 3.4.0
Resolution: Fixed
> Fix dependency error in hadoop-hdfs on M1 Mac
> -
>
> Key: HDFS-16523
> URL: https://issues.apache.org/jira/browse/HDFS-16523
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
> Environment: M1 Pro Mac
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> hadoop-hdfs build is failing on docker with M1 Mac.
> {code}
> [WARNING]
> Dependency convergence error for
> org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided paths to
> dependency are:
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
> +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.openlabtesting.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided
> and
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
> +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.fusesource.leveldbjni:leveldbjni-osx:jar:1.8:provided
> +-org.fusesource.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.9:provided
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16523) Fix dependency error in hadoop-hdfs on M1 Mac
[
https://issues.apache.org/jira/browse/HDFS-16523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17514159#comment-17514159
]
Steve Loughran commented on HDFS-16523:
---
merged into trunk; backport as needed.
> Fix dependency error in hadoop-hdfs on M1 Mac
> -
>
> Key: HDFS-16523
> URL: https://issues.apache.org/jira/browse/HDFS-16523
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
> Environment: M1 Pro Mac
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> hadoop-hdfs build is failing on docker with M1 Mac.
> {code}
> [WARNING]
> Dependency convergence error for
> org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided paths to
> dependency are:
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
> +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.openlabtesting.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided
> and
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
> +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.fusesource.leveldbjni:leveldbjni-osx:jar:1.8:provided
> +-org.fusesource.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.9:provided
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16523) Fix dependency error in hadoop-hdfs on M1 Mac
[
https://issues.apache.org/jira/browse/HDFS-16523?focusedWorklogId=749410&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749410
]
ASF GitHub Bot logged work on HDFS-16523:
-
Author: ASF GitHub Bot
Created on: 29/Mar/22 15:45
Start Date: 29/Mar/22 15:45
Worklog Time Spent: 10m
Work Description: steveloughran merged pull request #4112:
URL: https://github.com/apache/hadoop/pull/4112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 749410)
Time Spent: 0.5h (was: 20m)
> Fix dependency error in hadoop-hdfs on M1 Mac
> -
>
> Key: HDFS-16523
> URL: https://issues.apache.org/jira/browse/HDFS-16523
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
> Environment: M1 Pro Mac
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> hadoop-hdfs build is failing on docker with M1 Mac.
> {code}
> [WARNING]
> Dependency convergence error for
> org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided paths to
> dependency are:
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
> +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.openlabtesting.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.11:provided
> and
> +-org.apache.hadoop:hadoop-hdfs:jar:3.4.0-SNAPSHOT
> +-org.openlabtesting.leveldbjni:leveldbjni-all:jar:1.8:compile
> +-org.fusesource.leveldbjni:leveldbjni-osx:jar:1.8:provided
> +-org.fusesource.leveldbjni:leveldbjni:jar:1.8:provided
> +-org.fusesource.hawtjni:hawtjni-runtime:jar:1.9:provided
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16520) Improve EC pread: avoid potential reading whole block
[
https://issues.apache.org/jira/browse/HDFS-16520?focusedWorklogId=749392&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749392
]
ASF GitHub Bot logged work on HDFS-16520:
-
Author: ASF GitHub Bot
Created on: 29/Mar/22 15:21
Start Date: 29/Mar/22 15:21
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #4104:
URL: https://github.com/apache/hadoop/pull/4104#issuecomment-1082009485
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 19s | | trunk passed |
| +1 :green_heart: | compile | 6m 5s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 5m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 49s | | trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 16s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 41s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 7s | | the patch passed |
| +1 :green_heart: | compile | 5m 44s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 5m 44s | | the patch passed |
| +1 :green_heart: | compile | 5m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 5m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 2s |
[/results-checkstyle-hadoop-hdfs-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4104/3/artifact/out/results-checkstyle-hadoop-hdfs-project.txt)
| hadoop-hdfs-project: The patch generated 3 new + 29 unchanged - 0 fixed =
32 total (was 29) |
| +1 :green_heart: | mvnsite | 2m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 0s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 23s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 226m 1s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 361m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4104/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4104 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 259ffc61732c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3dac78493d53795dbe739c9942e9594b1b84da6a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
[jira] [Work logged] (HDFS-16518) KeyProviderCache close cached KeyProvider with Hadoop ShutdownHookManager
[
https://issues.apache.org/jira/browse/HDFS-16518?focusedWorklogId=749335&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-749335
]
ASF GitHub Bot logged work on HDFS-16518:
-
Author: ASF GitHub Bot
Created on: 29/Mar/22 13:41
Start Date: 29/Mar/22 13:41
Worklog Time Spent: 10m
Work Description: aajisaka commented on pull request #4100:
URL: https://github.com/apache/hadoop/pull/4100#issuecomment-1081888202
Hi @omalley - it is committed to trunk, branch-3.3, and branch-2.10 without
your approval in either JIRA or GitHub PR. Do you have any reason to approve
the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 749335)
Time Spent: 1h 50m (was: 1h 40m)
> KeyProviderCache close cached KeyProvider with Hadoop ShutdownHookManager
> -
>
> Key: HDFS-16518
> URL: https://issues.apache.org/jira/browse/HDFS-16518
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.10.0
>Reporter: Lei Yang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> KeyProvider implements Closable interface but some custom implementation of
> KeyProvider also needs explicit close in KeyProviderCache. An example is to
> use custom KeyProvider in DFSClient to integrate read encrypted file on HDFS.
> KeyProvider currently gets closed in KeyProviderCache only when cache entry
> is expired or invalidated. In some cases, this is not happening. This seems
> related to guava cache.
> This patch is to use hadoop JVM shutdownhookManager to globally cleanup cache
> entries and thus close KeyProvider using cache hook right after filesystem
> instance gets closed in a deterministic way.
> {code:java}
> Class KeyProviderCache
> ...
> public KeyProviderCache(long expiryMs) {
> cache = CacheBuilder.newBuilder()
> .expireAfterAccess(expiryMs, TimeUnit.MILLISECONDS)
> .removalListener(new RemovalListener() {
> @Override
> public void onRemoval(
> @Nonnull RemovalNotification notification) {
> try {
> assert notification.getValue() != null;
> notification.getValue().close();
> } catch (Throwable e) {
> LOG.error(
> "Error closing KeyProvider with uri ["
> + notification.getKey() + "]", e);
> }
> }
> })
> .build();
> }{code}
> We could have made a new function KeyProviderCache#close, have each DFSClient
> call this function and close KeyProvider at the end of each DFSClient#close
> call but it will expose another problem to potentially close global cache
> among different DFSClient instances.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
