[jira] [Created] (HDFS-16526) Add metrics for slow DataNode
Renukaprasad C created HDFS-16526: - Summary: Add metrics for slow DataNode Key: HDFS-16526 URL: https://issues.apache.org/jira/browse/HDFS-16526 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Renukaprasad C Assignee: Renukaprasad C Add some more metrics for slow datanode operations - FlushOrSync, PacketResponder send ACK. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16518) KeyProviderCache close cached KeyProvider with Hadoop ShutdownHookManager
[
https://issues.apache.org/jira/browse/HDFS-16518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley resolved HDFS-16518.
--
Fix Version/s: 3.4.0
2.10.2
3.3.3
Resolution: Fixed
I committed this. Thanks, Lei!
> KeyProviderCache close cached KeyProvider with Hadoop ShutdownHookManager
> -
>
> Key: HDFS-16518
> URL: https://issues.apache.org/jira/browse/HDFS-16518
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.10.0
>Reporter: Lei Yang
>Assignee: Lei Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 2.10.2, 3.3.3
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> KeyProvider implements Closable interface but some custom implementation of
> KeyProvider also needs explicit close in KeyProviderCache. An example is to
> use custom KeyProvider in DFSClient to integrate read encrypted file on HDFS.
> KeyProvider currently gets closed in KeyProviderCache only when cache entry
> is expired or invalidated. In some cases, this is not happening. This seems
> related to guava cache.
> This patch is to use hadoop JVM shutdownhookManager to globally cleanup cache
> entries and thus close KeyProvider using cache hook right after filesystem
> instance gets closed in a deterministic way.
> {code:java}
> Class KeyProviderCache
> ...
> public KeyProviderCache(long expiryMs) {
> cache = CacheBuilder.newBuilder()
> .expireAfterAccess(expiryMs, TimeUnit.MILLISECONDS)
> .removalListener(new RemovalListener() {
> @Override
> public void onRemoval(
> @Nonnull RemovalNotification notification) {
> try {
> assert notification.getValue() != null;
> notification.getValue().close();
> } catch (Throwable e) {
> LOG.error(
> "Error closing KeyProvider with uri ["
> + notification.getKey() + "]", e);
> }
> }
> })
> .build();
> }{code}
> We could have made a new function KeyProviderCache#close, have each DFSClient
> call this function and close KeyProvider at the end of each DFSClient#close
> call but it will expose another problem to potentially close global cache
> among different DFSClient instances.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/ [Mar 29, 2022 7:58:27 AM] (gyorandr) YARN-10549. Decouple RM runner logic from SLSRunner. Contributed by Szilard Nemeth. [Mar 29, 2022 3:44:55 PM] (noreply) HDFS-16523. Fix dependency error in hadoop-hdfs on M1 Mac (#4112) -1 overall The following subsystems voted -1: blanks pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-tools/hadoop-sls org.apache.hadoop.yarn.sls.AMRunner.setInputTraces(String[]) may expose internal representation by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:[line 268] Write to static field org.apache.hadoop.yarn.sls.AMRunner.REMAINING_APPS from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:[line 116] spotbugs : module:hadoop-tools org.apache.hadoop.yarn.sls.AMRunner.setInputTraces(String[]) may expose internal representation by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:[line 268] Write to static field org.apache.hadoop.yarn.sls.AMRunner.REMAINING_APPS from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:[line 116] spotbugs : module:root org.apache.hadoop.yarn.sls.AMRunner.setInputTraces(String[]) may expose internal representation by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:by storing an externally mutable object into AMRunner.inputTraces At AMRunner.java:[line 268] Write to static field org.apache.hadoop.yarn.sls.AMRunner.REMAINING_APPS from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:from instance method org.apache.hadoop.yarn.sls.AMRunner.startAM() At AMRunner.java:[line 116] Failed junit tests : hadoop.yarn.conf.TestYarnConfigurationFields cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/results-compile-javac-root.txt [340K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/blanks-eol.txt [13M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/results-checkstyle-root.txt [14M] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/results-shellcheck.txt [28K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/results-javadoc-javadoc-root.txt [400K] spotbugs: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/824/artifact/out/branch-spotbugs-hadoop-tools_hadoop-sls-warnings.html [8.0K]
[jira] [Resolved] (HDFS-16413) Reconfig dfs usage parameters for datanode
[ https://issues.apache.org/jira/browse/HDFS-16413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma resolved HDFS-16413. - Fix Version/s: 3.4.0 Resolution: Fixed > Reconfig dfs usage parameters for datanode > -- > > Key: HDFS-16413 > URL: https://issues.apache.org/jira/browse/HDFS-16413 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Reconfig dfs usage parameters for datanode. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16525) System.err should be used when error occurs in multiple methods in DFSAdmin class
yanbin.zhang created HDFS-16525: --- Summary: System.err should be used when error occurs in multiple methods in DFSAdmin class Key: HDFS-16525 URL: https://issues.apache.org/jira/browse/HDFS-16525 Project: Hadoop HDFS Issue Type: Bug Components: dfsadmin Affects Versions: 3.3.2 Reporter: yanbin.zhang Assignee: yanbin.zhang -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Call for Presentations now open, ApacheCon North America 2022
[You are receiving this because you are subscribed to one or more user or dev mailing list of an Apache Software Foundation project.] ApacheCon draws participants at all levels to explore “Tomorrow’s Technology Today” across 300+ Apache projects and their diverse communities. ApacheCon showcases the latest developments in ubiquitous Apache projects and emerging innovations through hands-on sessions, keynotes, real-world case studies, trainings, hackathons, community events, and more. The Apache Software Foundation will be holding ApacheCon North America 2022 at the New Orleans Sheration, October 3rd through 6th, 2022. The Call for Presentations is now open, and will close at 00:01 UTC on May 23rd, 2022. We are accepting presentation proposals for any topic that is related to the Apache mission of producing free software for the public good. This includes, but is not limited to: Community Big Data Search IoT Cloud Fintech Pulsar Tomcat You can submit your session proposals starting today at https://cfp.apachecon.com/ Rich Bowen, on behalf of the ApacheCon Planners apachecon.com @apachecon - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/ No changes -1 overall The following subsystems voted -1: asflicense hadolint mvnsite pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.fs.TestFileUtil hadoop.io.compress.snappy.TestSnappyCompressorDecompressor hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat hadoop.hdfs.server.federation.router.TestRouterQuota hadoop.hdfs.server.federation.resolver.order.TestLocalResolver hadoop.yarn.server.resourcemanager.TestClientRMService hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter hadoop.mapreduce.lib.input.TestLineRecordReader hadoop.mapred.TestLineRecordReader hadoop.yarn.sls.TestSLSRunner hadoop.resourceestimator.service.TestResourceEstimatorService hadoop.resourceestimator.solver.impl.TestLpSolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/diff-compile-javac-root.txt [476K] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/diff-checkstyle-root.txt [14M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/diff-patch-hadolint.txt [4.0K] mvnsite: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-mvnsite-root.txt [560K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/pathlen.txt [12K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/diff-patch-shellcheck.txt [72K] whitespace: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/whitespace-eol.txt [12M] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-javadoc-root.txt [40K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [224K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [428K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt [12K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [36K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [128K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt [104K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-unit-hadoop-tools_hadoop-azure.txt [20K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-unit-hadoop-tools_hadoop-sls.txt [28K] https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/616/artifact/out/patch-unit-hadoop-tools_hadoop-resourceestimator.txt [16K] asflicense:
[jira] [Resolved] (HDFS-16498) Fix NPE for checkBlockReportLease
[ https://issues.apache.org/jira/browse/HDFS-16498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-16498. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to trunk. Thanks @tomscut for your contribution! > Fix NPE for checkBlockReportLease > - > > Key: HDFS-16498 > URL: https://issues.apache.org/jira/browse/HDFS-16498 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2022-03-09-20-35-22-028.png, screenshot-1.png > > Time Spent: 5h 20m > Remaining Estimate: 0h > > During the restart of Namenode, a Datanode is not registered, but this > Datanode triggers FBR, which causes NPE. > !image-2022-03-09-20-35-22-028.png|width=871,height=158! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
