[jira] [Resolved] (HDFS-16921) The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy.
[ https://issues.apache.org/jira/browse/HDFS-16921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangHB resolved HDFS-16921. Resolution: Duplicate > The logic of IncrementalBlockReportManager#addRDBI method may cause missing > blocks when cluster is busy. > > > Key: HDFS-16921 > URL: https://issues.apache.org/jira/browse/HDFS-16921 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.3.4 >Reporter: ZhangHB >Priority: Critical > > The current logic of IncrementalBlockReportManager# addRDBI method could lead > to the missing blocks when datanodes in pipeline are I/O busy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16920) The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy.
[ https://issues.apache.org/jira/browse/HDFS-16920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangHB resolved HDFS-16920. Resolution: Duplicate > The logic of IncrementalBlockReportManager#addRDBI method may cause missing > blocks when cluster is busy. > > > Key: HDFS-16920 > URL: https://issues.apache.org/jira/browse/HDFS-16920 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.3.4 >Reporter: ZhangHB >Priority: Critical > > The current logic of IncrementalBlockReportManager# addRDBI method could lead > to the missing blocks when datanodes in pipeline are I/O busy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16919) The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy.
[ https://issues.apache.org/jira/browse/HDFS-16919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZhangHB resolved HDFS-16919. Resolution: Duplicate > The logic of IncrementalBlockReportManager#addRDBI method may cause missing > blocks when cluster is busy. > > > Key: HDFS-16919 > URL: https://issues.apache.org/jira/browse/HDFS-16919 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.3.4 >Reporter: ZhangHB >Priority: Critical > > The current logic of IncrementalBlockReportManager# addRDBI method could lead > to the missing blocks when datanodes in pipeline are I/O busy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16922) The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy.
ZhangHB created HDFS-16922: -- Summary: The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy. Key: HDFS-16922 URL: https://issues.apache.org/jira/browse/HDFS-16922 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: ZhangHB The current logic of IncrementalBlockReportManager# addRDBI method could lead to the missing blocks when datanodes in pipeline are I/O busy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16919) The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy.
ZhangHB created HDFS-16919: -- Summary: The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy. Key: HDFS-16919 URL: https://issues.apache.org/jira/browse/HDFS-16919 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.3.4 Reporter: ZhangHB The current logic of IncrementalBlockReportManager# addRDBI method could lead to the missing blocks when datanodes in pipeline are I/O busy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16921) The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy.
ZhangHB created HDFS-16921: -- Summary: The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy. Key: HDFS-16921 URL: https://issues.apache.org/jira/browse/HDFS-16921 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.3.4 Reporter: ZhangHB The current logic of IncrementalBlockReportManager# addRDBI method could lead to the missing blocks when datanodes in pipeline are I/O busy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16920) The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy.
ZhangHB created HDFS-16920: -- Summary: The logic of IncrementalBlockReportManager#addRDBI method may cause missing blocks when cluster is busy. Key: HDFS-16920 URL: https://issues.apache.org/jira/browse/HDFS-16920 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.3.4 Reporter: ZhangHB The current logic of IncrementalBlockReportManager# addRDBI method could lead to the missing blocks when datanodes in pipeline are I/O busy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16918) Optionally shut down datanode if it does not stay connected to active namenode
Viraj Jasani created HDFS-16918: --- Summary: Optionally shut down datanode if it does not stay connected to active namenode Key: HDFS-16918 URL: https://issues.apache.org/jira/browse/HDFS-16918 Project: Hadoop HDFS Issue Type: New Feature Reporter: Viraj Jasani Assignee: Viraj Jasani While deploying Hdfs on Envoy proxy setup, depending on the socket timeout configured at envoy, the network connection issues or packet loss could be observed. All of envoys basically form a transparent communication mesh in which each app can send and receive packets to and from localhost and is unaware of the network topology. The primary purpose of Envoy is to make the network transparent to applications, in order to identify network issues reliably. However, sometimes such proxy based setup could result into socket connection issues b/ datanode and namenode. Many deployment frameworks provide auto-start functionality when any of the hadoop daemons are stopped. If a given datanode does not stay connected to active namenode in the cluster i.e. does not receive heartbeat response in time from active namenode (even though active namenode is not terminated), it would not be much useful. We should be able to provide configurable behavior such that if a given datanode cannot receive heartbeat response from active namenode in configurable time duration, it should terminate itself to avoid impacting the availability SLA. This is specifically helpful when the underlying deployment or observability framework (e.g. K8S) can start up the datanode automatically upon it's shutdown (unless it is being restarted as part of rolling upgrade) and help the newly brought up datanode (in case of k8s, a new pod with dynamically changing nodes) establish new socket connection to active and standby namenodes. This should be an opt-in behavior and not default one. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16917) Add transfer rate quantile metrics for DataNode reads
Ravindra Dingankar created HDFS-16917: - Summary: Add transfer rate quantile metrics for DataNode reads Key: HDFS-16917 URL: https://issues.apache.org/jira/browse/HDFS-16917 Project: Hadoop HDFS Issue Type: Task Components: datanode Reporter: Ravindra Dingankar Currently we have the following metrics for datanode reads. |BytesRead BlocksRead TotalReadTime|Total number of bytes read from DataNode Total number of blocks read from DataNode Total number of milliseconds spent on read operation| We would like to add a new quantile metric calculating the distribution of data transfer rate for datanode reads. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: Request to create an ASF Jira account
I've submitted a request to create the account. You should receive an email shortly. On Tue, Feb 14, 2023 at 1:40 PM ravindra dingankar < ravindradingan...@gmail.com> wrote: > Hi, > > I am part of LinkedIn's HDFS team, and would like to start contributing to > HDFS and be part of the mailing list. > > I request the project to create an ASF Jira account for me. > My details are as follows > > email address : ravindra.dingan...@asu.edu > preferred username : rdingankar > alternate username : rdingank > display name : Ravindra Dingankar > > > Thanks & Regards, > Ravindra Dingankar >
Request to create an ASF Jira account
Hi, I am part of LinkedIn's HDFS team, and would like to start contributing to HDFS and be part of the mailing list. I request the project to create an ASF Jira account for me. My details are as follows email address : ravindra.dingan...@asu.edu preferred username : rdingankar alternate username : rdingank display name : Ravindra Dingankar Thanks & Regards, Ravindra Dingankar
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1136/ [Feb 13, 2023, 2:31:32 PM] (github) HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5372) [Feb 13, 2023, 3:21:27 PM] (github) MAPREDUCE-7433. Remove unused mapred/LoggingHttpResponseEncoder.java. (#5388) [Feb 13, 2023, 6:33:02 PM] (github) HADOOP-18524. Deploy Hadoop trunk version website. (#5386). Contributed by Ayush Saxena. [Feb 13, 2023, 7:24:06 PM] (github) HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) -1 overall The following subsystems voted -1: blanks hadolint pathlen spotbugs unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: XML : Parsing Error(s): hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml spotbugs : module:hadoop-mapreduce-project/hadoop-mapreduce-client Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:hadoop-mapreduce-project Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] spotbugs : module:root Write to static field org.apache.hadoop.mapreduce.task.reduce.Fetcher.nextId from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:from instance method new org.apache.hadoop.mapreduce.task.reduce.Fetcher(JobConf, TaskAttemptID, ShuffleSchedulerImpl, MergeManager, Reporter, ShuffleClientMetrics, ExceptionReporter, SecretKey) At Fetcher.java:[line 120] Failed junit tests : hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestJobHistoryParsing hadoop.mapreduce.v2.hs.TestJobHistoryEvents hadoop.mapreduce.v2.hs.TestJobHistoryServer hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp cc: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1136/artifact/out/results-compile-cc-root.txt [96K] javac: https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1136/artifact/out/results-compile-javac-root.txt [528K] blanks:
[jira] [Created] (HDFS-16916) Improve the use of JUnit Test in DFSClient
Hualong Zhang created HDFS-16916: Summary: Improve the use of JUnit Test in DFSClient Key: HDFS-16916 URL: https://issues.apache.org/jira/browse/HDFS-16916 Project: Hadoop HDFS Issue Type: Improvement Components: dfsclient Affects Versions: 3.4.0 Reporter: Hualong Zhang Improve the use of JUnit Test in DFSClient -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-16915) Optimize metrics for operations hold lock times of FsDatasetImpl
ZhangHB created HDFS-16915: -- Summary: Optimize metrics for operations hold lock times of FsDatasetImpl Key: HDFS-16915 URL: https://issues.apache.org/jira/browse/HDFS-16915 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.3.4 Reporter: ZhangHB Current calculation method also includes the time of waiting lock. So, i think we should optimize the compute method of metrics for operations hold lock times of FsDatasetImpl. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
