[jira] [Resolved] (HDFS-17055) Export HAState as a metric from Namenode for monitoring
[
https://issues.apache.org/jira/browse/HDFS-17055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri resolved HDFS-17055.
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Export HAState as a metric from Namenode for monitoring
> ---
>
> Key: HDFS-17055
> URL: https://issues.apache.org/jira/browse/HDFS-17055
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.4.0, 3.3.9
>Reporter: Xing Lin
>Assignee: Xing Lin
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> We'd like measure the uptime for Namenodes: percentage of time when we have
> the active/standby/observer node available (up and running). We could monitor
> the namenode from an external service, such as ZKFC. But that would require
> the external service to be available 100% itself. And when this third-party
> external monitoring service is down, we won't have info on whether our
> Namenodes are still up.
> We propose to take a different approach: we will emit Namenode state directly
> from namenode itself. Whenever we miss a data point for this metric, we
> consider the corresponding namenode to be down/not available. In other words,
> we assume the metric collection/monitoring infrastructure to be 100% reliable.
> One implementation detail: in hadoop, we have the _NameNodeMetrics_ class,
> which is currently used to emit all metrics for {_}NameNode.java{_}. However,
> we don't think that is a good place to emit NameNode HAState. HAState is
> stored in NameNode.java and we should directly emit it from NameNode.java.
> Otherwise, we basically duplicate this info in two classes and we would have
> to keep them in sync. Besides, _NameNodeMetrics_ class does not have a
> reference to the _NameNode_ object which it belongs to. An _NameNodeMetrics_
> is created by a _static_ function _initMetrics()_ in {_}NameNode.java{_}.
> We shouldn't emit HA state from FSNameSystem.java either, as it is
> initialized from NameNode.java and all state transitions are implemented in
> NameNode.java.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/ No changes -1 overall The following subsystems voted -1: blanks hadolint pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml Failed junit tests : hadoop.mapreduce.v2.TestUberAM hadoop.mapreduce.v2.TestMRJobsWithProfiler hadoop.mapreduce.v2.TestMRJobs hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/results-compile-javac-root.txt [12K] blanks: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/blanks-eol.txt [14M] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/blanks-tabs.txt [2.0M] checkstyle: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/results-checkstyle-root.txt [13M] hadolint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/results-hadolint.txt [20K] pathlen: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/results-pathlen.txt [16K] pylint: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/results-pylint.txt [20K] shellcheck: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/results-shellcheck.txt [24K] xml: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/xml.txt [24K] javadoc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/results-javadoc-javadoc-root.txt [244K] unit: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt [72K] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1269/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt [96K] Powered by Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[ANNOUNCE] Apache Hadoop 3.3.6 release
On behalf of the Apache Hadoop Project Management Committee, I am pleased to announce the release of Apache Hadoop 3.3.6. It contains 117 bug fixes, improvements and enhancements since 3.3.5. Users of Apache Hadoop 3.3.5 and earlier should upgrade to this release. https://hadoop.apache.org/release/3.3.6.html Feature highlights: SBOM artifacts Starting from this release, Hadoop publishes Software Bill of Materials (SBOM) using CycloneDX Maven plugin. For more information about SBOM, please go to [SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM). HDFS RBF: RDBMS based token storage support HDFS Router-Router Based Federation now supports storing delegation tokens on MySQL, [HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535) which improves token operation through over the original Zookeeper-based implementation. New File System APIs [HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved a number of HDFS-specific APIs to Hadoop Common to make it possible for certain applications that depend on HDFS semantics to run on other Hadoop compatible file systems. In particular, recoverLease() and isFileClosed() are exposed through LeaseRecoverable interface. While setSafeMode() is exposed through SafeMode interface. Many thanks to everyone who helped in this release by supplying patches, reviewing them, helping get this release building and testing and reviewing the final artifacts. Weichiu
[jira] [Resolved] (HDFS-17057) RBF: Add DataNode maintenance states to Federation UI
[ https://issues.apache.org/jira/browse/HDFS-17057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved HDFS-17057. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > RBF: Add DataNode maintenance states to Federation UI > -- > > Key: HDFS-17057 > URL: https://issues.apache.org/jira/browse/HDFS-17057 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haiyang Hu >Assignee: Haiyang Hu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Add DataNode maintenance states to Federation UI -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
