[jira] [Updated] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-17434: Description: In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges from 5ms to 10ms, exceeding the usual disk reading overhead. Our machine network card bandwidth is 2Mb/s. !image-2024-03-20-19-10-13-016.png|width=662,height=135! !image-2024-03-20-19-55-18-378.png! By adding log printing, it turns out that the Selector.select function has significant overhead. !image-2024-03-20-19-22-29-829.png|width=474,height=262! !image-2024-03-20-19-24-02-233.png|width=445,height=181! I would like to know if this falls within the normal range or how we can improve it. was: In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges from 5ms to 10ms, exceeding the usual disk reading overhead. Our machine network card bandwidth is 10Gb/s. !image-2024-03-20-19-10-13-016.png|width=662,height=135! By adding log printing, it turns out that the Selector.select function has significant overhead. !image-2024-03-20-19-22-29-829.png|width=474,height=262! !image-2024-03-20-19-24-02-233.png|width=445,height=181! I would like to know if this falls within the normal range or how we can improve it. > Selector.select in SocketIOWithTimeout.java has significant overhead > > > Key: HDFS-17434 > URL: https://issues.apache.org/jira/browse/HDFS-17434 > Project: Hadoop HDFS > Issue Type: Test >Reporter: qinyuren >Priority: Major > Attachments: image-2024-03-20-19-10-13-016.png, > image-2024-03-20-19-22-29-829.png, image-2024-03-20-19-24-02-233.png, > image-2024-03-20-19-55-18-378.png > > > In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges > from 5ms to 10ms, exceeding the usual disk reading overhead. Our machine > network card bandwidth is 2Mb/s. > !image-2024-03-20-19-10-13-016.png|width=662,height=135! > !image-2024-03-20-19-55-18-378.png! > By adding log printing, it turns out that the Selector.select function has > significant overhead. > !image-2024-03-20-19-22-29-829.png|width=474,height=262! > !image-2024-03-20-19-24-02-233.png|width=445,height=181! > I would like to know if this falls within the normal range or how we can > improve it. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-17434: Issue Type: Wish (was: Task) > Selector.select in SocketIOWithTimeout.java has significant overhead > > > Key: HDFS-17434 > URL: https://issues.apache.org/jira/browse/HDFS-17434 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: qinyuren >Priority: Major > Attachments: image-2024-03-20-19-10-13-016.png, > image-2024-03-20-19-22-29-829.png, image-2024-03-20-19-24-02-233.png > > > In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges > from 5ms to 10ms, exceeding the usual disk reading overhead. Our machine > network card bandwidth is 10Gb/s. > !image-2024-03-20-19-10-13-016.png|width=662,height=135! > By adding log printing, it turns out that the Selector.select function has > significant overhead. > !image-2024-03-20-19-22-29-829.png|width=474,height=262! > !image-2024-03-20-19-24-02-233.png|width=445,height=181! > I would like to know if this falls within the normal range or how we can > improve it. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828925#comment-17828925 ] qinyuren commented on HDFS-17434: - [~hexiaoqiao] [~tasanuma] [~zanderxu] Please take a look. > Selector.select in SocketIOWithTimeout.java has significant overhead > > > Key: HDFS-17434 > URL: https://issues.apache.org/jira/browse/HDFS-17434 > Project: Hadoop HDFS > Issue Type: Task >Reporter: qinyuren >Priority: Major > Attachments: image-2024-03-20-19-10-13-016.png, > image-2024-03-20-19-22-29-829.png, image-2024-03-20-19-24-02-233.png > > > In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges > from 5ms to 10ms, exceeding the usual disk reading overhead. Our machine > network card bandwidth is 10Gb/s. > !image-2024-03-20-19-10-13-016.png|width=662,height=135! > By adding log printing, it turns out that the Selector.select function has > significant overhead. > !image-2024-03-20-19-22-29-829.png|width=474,height=262! > !image-2024-03-20-19-24-02-233.png|width=445,height=181! > I would like to know if this falls within the normal range or how we can > improve it. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-17434: Attachment: image-2024-03-20-19-55-18-378.png > Selector.select in SocketIOWithTimeout.java has significant overhead > > > Key: HDFS-17434 > URL: https://issues.apache.org/jira/browse/HDFS-17434 > Project: Hadoop HDFS > Issue Type: Test >Reporter: qinyuren >Priority: Major > Attachments: image-2024-03-20-19-10-13-016.png, > image-2024-03-20-19-22-29-829.png, image-2024-03-20-19-24-02-233.png, > image-2024-03-20-19-55-18-378.png > > > In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges > from 5ms to 10ms, exceeding the usual disk reading overhead. Our machine > network card bandwidth is 10Gb/s. > !image-2024-03-20-19-10-13-016.png|width=662,height=135! > By adding log printing, it turns out that the Selector.select function has > significant overhead. > !image-2024-03-20-19-22-29-829.png|width=474,height=262! > !image-2024-03-20-19-24-02-233.png|width=445,height=181! > I would like to know if this falls within the normal range or how we can > improve it. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-17434: Issue Type: Test (was: Wish) > Selector.select in SocketIOWithTimeout.java has significant overhead > > > Key: HDFS-17434 > URL: https://issues.apache.org/jira/browse/HDFS-17434 > Project: Hadoop HDFS > Issue Type: Test >Reporter: qinyuren >Priority: Major > Attachments: image-2024-03-20-19-10-13-016.png, > image-2024-03-20-19-22-29-829.png, image-2024-03-20-19-24-02-233.png > > > In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges > from 5ms to 10ms, exceeding the usual disk reading overhead. Our machine > network card bandwidth is 10Gb/s. > !image-2024-03-20-19-10-13-016.png|width=662,height=135! > By adding log printing, it turns out that the Selector.select function has > significant overhead. > !image-2024-03-20-19-22-29-829.png|width=474,height=262! > !image-2024-03-20-19-24-02-233.png|width=445,height=181! > I would like to know if this falls within the normal range or how we can > improve it. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead
[ https://issues.apache.org/jira/browse/HDFS-17434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-17434: Description: In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges from 5ms to 10ms, exceeding the usual disk reading overhead. Our machine network card bandwidth is 10Gb/s. !image-2024-03-20-19-10-13-016.png|width=662,height=135! By adding log printing, it turns out that the Selector.select function has significant overhead. !image-2024-03-20-19-22-29-829.png|width=474,height=262! !image-2024-03-20-19-24-02-233.png|width=445,height=181! I would like to know if this falls within the normal range or how we can improve it. was: In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges from 5ms to 10ms, exceeding the usual disk reading overhead. !image-2024-03-20-19-10-13-016.png|width=662,height=135! By adding log printing, it turns out that the Selector.select function has significant overhead. !image-2024-03-20-19-22-29-829.png|width=474,height=262! !image-2024-03-20-19-24-02-233.png|width=445,height=181! I would like to know if this falls within the normal range or how we can improve it. > Selector.select in SocketIOWithTimeout.java has significant overhead > > > Key: HDFS-17434 > URL: https://issues.apache.org/jira/browse/HDFS-17434 > Project: Hadoop HDFS > Issue Type: Task >Reporter: qinyuren >Priority: Major > Attachments: image-2024-03-20-19-10-13-016.png, > image-2024-03-20-19-22-29-829.png, image-2024-03-20-19-24-02-233.png > > > In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges > from 5ms to 10ms, exceeding the usual disk reading overhead. Our machine > network card bandwidth is 10Gb/s. > !image-2024-03-20-19-10-13-016.png|width=662,height=135! > By adding log printing, it turns out that the Selector.select function has > significant overhead. > !image-2024-03-20-19-22-29-829.png|width=474,height=262! > !image-2024-03-20-19-24-02-233.png|width=445,height=181! > I would like to know if this falls within the normal range or how we can > improve it. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-17434) Selector.select in SocketIOWithTimeout.java has significant overhead
qinyuren created HDFS-17434: --- Summary: Selector.select in SocketIOWithTimeout.java has significant overhead Key: HDFS-17434 URL: https://issues.apache.org/jira/browse/HDFS-17434 Project: Hadoop HDFS Issue Type: Task Reporter: qinyuren Attachments: image-2024-03-20-19-10-13-016.png, image-2024-03-20-19-22-29-829.png, image-2024-03-20-19-24-02-233.png In our cluster, the SendDataPacketBlockedOnNetworkNanosAvgTime metric ranges from 5ms to 10ms, exceeding the usual disk reading overhead. !image-2024-03-20-19-10-13-016.png|width=662,height=135! By adding log printing, it turns out that the Selector.select function has significant overhead. !image-2024-03-20-19-22-29-829.png|width=474,height=262! !image-2024-03-20-19-24-02-233.png|width=445,height=181! I would like to know if this falls within the normal range or how we can improve it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15413) DFSStripedInputStream throws exception when datanodes close idle connections
[
https://issues.apache.org/jira/browse/HDFS-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783445#comment-17783445
]
qinyuren edited comment on HDFS-15413 at 11/7/23 1:46 AM:
--
We also encountered datanode socket timeout problems in the process of using
EC. That's because our dfsclient has a lot concurrent read requests, and the
default value of *stripedReadThreadpoolSize* is 18, so we adjusted the value
of config *dfs.client.read.striped.threadpool.size* to solve the problem.
I hope it was helpful.
was (Author: qinyuren):
We also encountered datanode socket timeout problems in the process of using EC,
That's because our dfsclient has a lot concurrent read requests, and the
default value of
*stripedReadThreadpoolSize* is 18, so we adjusted the value of config
*dfs.client.read.striped.threadpool.size* to solve the problem.
I hope it was helpful.
> DFSStripedInputStream throws exception when datanodes close idle connections
>
>
> Key: HDFS-15413
> URL: https://issues.apache.org/jira/browse/HDFS-15413
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, erasure-coding, hdfs-client
>Affects Versions: 3.1.3
> Environment: - Hadoop 3.1.3
> - erasure coding with ISA-L and RS-3-2-1024k scheme
> - running in kubernetes
> - dfs.client.socket-timeout = 1
> - dfs.datanode.socket.write.timeout = 1
>Reporter: Andrey Elenskiy
>Priority: Critical
> Labels: pull-request-available
> Attachments: out.log
>
>
> We've run into an issue with compactions failing in HBase when erasure coding
> is enabled on a table directory. After digging further I was able to narrow
> it down to a seek + read logic and able to reproduce the issue with hdfs
> client only:
> {code:java}
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.FSDataInputStream;
> public class ReaderRaw {
> public static void main(final String[] args) throws Exception {
> Path p = new Path(args[0]);
> int bufLen = Integer.parseInt(args[1]);
> int sleepDuration = Integer.parseInt(args[2]);
> int countBeforeSleep = Integer.parseInt(args[3]);
> int countAfterSleep = Integer.parseInt(args[4]);
> Configuration conf = new Configuration();
> FSDataInputStream istream = FileSystem.get(conf).open(p);
> byte[] buf = new byte[bufLen];
> int readTotal = 0;
> int count = 0;
> try {
> while (true) {
> istream.seek(readTotal);
> int bytesRemaining = bufLen;
> int bufOffset = 0;
> while (bytesRemaining > 0) {
> int nread = istream.read(buf, 0, bufLen);
> if (nread < 0) {
> throw new Exception("nread is less than zero");
> }
> readTotal += nread;
> bufOffset += nread;
> bytesRemaining -= nread;
> }
> count++;
> if (count == countBeforeSleep) {
> System.out.println("sleeping for " + sleepDuration + "
> milliseconds");
> Thread.sleep(sleepDuration);
> System.out.println("resuming");
> }
> if (count == countBeforeSleep + countAfterSleep) {
> System.out.println("done");
> break;
> }
> }
> } catch (Exception e) {
> System.out.println("exception on read " + count + " read total "
> + readTotal);
> throw e;
> }
> }
> }
> {code}
> The issue appears to be due to the fact that datanodes close the connection
> of EC client if it doesn't fetch next packet for longer than
> dfs.client.socket-timeout. The EC client doesn't retry and instead assumes
> that those datanodes went away resulting in "missing blocks" exception.
> I was able to consistently reproduce with the following arguments:
> {noformat}
> bufLen = 100 (just below 1MB which is the size of the stripe)
> sleepDuration = (dfs.client.socket-timeout + 1) * 1000 (in our case 11000)
> countBeforeSleep = 1
> countAfterSleep = 7
> {noformat}
> I've attached the entire log output of running the snippet above against
> erasure coded file with RS-3-2-1024k policy. And here are the logs from
> datanodes of disconnecting the client:
> datanode 1:
> {noformat}
> 2020-06-15 19:06:20,697 INFO datanode.DataNode: Likely the client has stopped
> reading, disconnecting it (datanode-v11-0-hadoop.hadoop:9866:DataXceiver
> error processing READ_BLOCK operation src: /10.128.23.40:53748 dst:
> /10.128.14.46:9866); java.net.SocketTimeoutException: 1 milli
[jira] [Comment Edited] (HDFS-15413) DFSStripedInputStream throws exception when datanodes close idle connections
[
https://issues.apache.org/jira/browse/HDFS-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783445#comment-17783445
]
qinyuren edited comment on HDFS-15413 at 11/7/23 1:45 AM:
--
We also encountered datanode socket timeout problems in the process of using EC,
That's because our dfsclient has a lot concurrent read requests, and the
default value of
*stripedReadThreadpoolSize* is 18, so we adjusted the value of config
*dfs.client.read.striped.threadpool.size* to solve the problem.
I hope it was helpful.
was (Author: qinyuren):
We also encountered datanode socket timeout problems in the process of using EC,
That's because our dfsclient has a lot concurrent read requests, and the
default value of *stripedReadThreadpoolSize* is 18, so we adjusted the value
of config *dfs.client.read.striped.threadpool.size* to solve the problem.
I hope it was helpful.
> DFSStripedInputStream throws exception when datanodes close idle connections
>
>
> Key: HDFS-15413
> URL: https://issues.apache.org/jira/browse/HDFS-15413
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, erasure-coding, hdfs-client
>Affects Versions: 3.1.3
> Environment: - Hadoop 3.1.3
> - erasure coding with ISA-L and RS-3-2-1024k scheme
> - running in kubernetes
> - dfs.client.socket-timeout = 1
> - dfs.datanode.socket.write.timeout = 1
>Reporter: Andrey Elenskiy
>Priority: Critical
> Labels: pull-request-available
> Attachments: out.log
>
>
> We've run into an issue with compactions failing in HBase when erasure coding
> is enabled on a table directory. After digging further I was able to narrow
> it down to a seek + read logic and able to reproduce the issue with hdfs
> client only:
> {code:java}
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.FSDataInputStream;
> public class ReaderRaw {
> public static void main(final String[] args) throws Exception {
> Path p = new Path(args[0]);
> int bufLen = Integer.parseInt(args[1]);
> int sleepDuration = Integer.parseInt(args[2]);
> int countBeforeSleep = Integer.parseInt(args[3]);
> int countAfterSleep = Integer.parseInt(args[4]);
> Configuration conf = new Configuration();
> FSDataInputStream istream = FileSystem.get(conf).open(p);
> byte[] buf = new byte[bufLen];
> int readTotal = 0;
> int count = 0;
> try {
> while (true) {
> istream.seek(readTotal);
> int bytesRemaining = bufLen;
> int bufOffset = 0;
> while (bytesRemaining > 0) {
> int nread = istream.read(buf, 0, bufLen);
> if (nread < 0) {
> throw new Exception("nread is less than zero");
> }
> readTotal += nread;
> bufOffset += nread;
> bytesRemaining -= nread;
> }
> count++;
> if (count == countBeforeSleep) {
> System.out.println("sleeping for " + sleepDuration + "
> milliseconds");
> Thread.sleep(sleepDuration);
> System.out.println("resuming");
> }
> if (count == countBeforeSleep + countAfterSleep) {
> System.out.println("done");
> break;
> }
> }
> } catch (Exception e) {
> System.out.println("exception on read " + count + " read total "
> + readTotal);
> throw e;
> }
> }
> }
> {code}
> The issue appears to be due to the fact that datanodes close the connection
> of EC client if it doesn't fetch next packet for longer than
> dfs.client.socket-timeout. The EC client doesn't retry and instead assumes
> that those datanodes went away resulting in "missing blocks" exception.
> I was able to consistently reproduce with the following arguments:
> {noformat}
> bufLen = 100 (just below 1MB which is the size of the stripe)
> sleepDuration = (dfs.client.socket-timeout + 1) * 1000 (in our case 11000)
> countBeforeSleep = 1
> countAfterSleep = 7
> {noformat}
> I've attached the entire log output of running the snippet above against
> erasure coded file with RS-3-2-1024k policy. And here are the logs from
> datanodes of disconnecting the client:
> datanode 1:
> {noformat}
> 2020-06-15 19:06:20,697 INFO datanode.DataNode: Likely the client has stopped
> reading, disconnecting it (datanode-v11-0-hadoop.hadoop:9866:DataXceiver
> error processing READ_BLOCK operation src: /10.128.23.40:53748 dst:
> /10.128.14.46:9866); java.net.SocketTimeoutException: 1 milli
[jira] [Commented] (HDFS-15413) DFSStripedInputStream throws exception when datanodes close idle connections
[
https://issues.apache.org/jira/browse/HDFS-15413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783445#comment-17783445
]
qinyuren commented on HDFS-15413:
-
We also encountered datanode socket timeout problems in the process of using EC,
That's because our dfsclient has a lot concurrent read requests, and the
default value of *stripedReadThreadpoolSize* is 18, so we adjusted the value
of config *dfs.client.read.striped.threadpool.size* to solve the problem.
I hope it was helpful.
> DFSStripedInputStream throws exception when datanodes close idle connections
>
>
> Key: HDFS-15413
> URL: https://issues.apache.org/jira/browse/HDFS-15413
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, erasure-coding, hdfs-client
>Affects Versions: 3.1.3
> Environment: - Hadoop 3.1.3
> - erasure coding with ISA-L and RS-3-2-1024k scheme
> - running in kubernetes
> - dfs.client.socket-timeout = 1
> - dfs.datanode.socket.write.timeout = 1
>Reporter: Andrey Elenskiy
>Priority: Critical
> Labels: pull-request-available
> Attachments: out.log
>
>
> We've run into an issue with compactions failing in HBase when erasure coding
> is enabled on a table directory. After digging further I was able to narrow
> it down to a seek + read logic and able to reproduce the issue with hdfs
> client only:
> {code:java}
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.fs.Path;
> import org.apache.hadoop.fs.FileSystem;
> import org.apache.hadoop.fs.FSDataInputStream;
> public class ReaderRaw {
> public static void main(final String[] args) throws Exception {
> Path p = new Path(args[0]);
> int bufLen = Integer.parseInt(args[1]);
> int sleepDuration = Integer.parseInt(args[2]);
> int countBeforeSleep = Integer.parseInt(args[3]);
> int countAfterSleep = Integer.parseInt(args[4]);
> Configuration conf = new Configuration();
> FSDataInputStream istream = FileSystem.get(conf).open(p);
> byte[] buf = new byte[bufLen];
> int readTotal = 0;
> int count = 0;
> try {
> while (true) {
> istream.seek(readTotal);
> int bytesRemaining = bufLen;
> int bufOffset = 0;
> while (bytesRemaining > 0) {
> int nread = istream.read(buf, 0, bufLen);
> if (nread < 0) {
> throw new Exception("nread is less than zero");
> }
> readTotal += nread;
> bufOffset += nread;
> bytesRemaining -= nread;
> }
> count++;
> if (count == countBeforeSleep) {
> System.out.println("sleeping for " + sleepDuration + "
> milliseconds");
> Thread.sleep(sleepDuration);
> System.out.println("resuming");
> }
> if (count == countBeforeSleep + countAfterSleep) {
> System.out.println("done");
> break;
> }
> }
> } catch (Exception e) {
> System.out.println("exception on read " + count + " read total "
> + readTotal);
> throw e;
> }
> }
> }
> {code}
> The issue appears to be due to the fact that datanodes close the connection
> of EC client if it doesn't fetch next packet for longer than
> dfs.client.socket-timeout. The EC client doesn't retry and instead assumes
> that those datanodes went away resulting in "missing blocks" exception.
> I was able to consistently reproduce with the following arguments:
> {noformat}
> bufLen = 100 (just below 1MB which is the size of the stripe)
> sleepDuration = (dfs.client.socket-timeout + 1) * 1000 (in our case 11000)
> countBeforeSleep = 1
> countAfterSleep = 7
> {noformat}
> I've attached the entire log output of running the snippet above against
> erasure coded file with RS-3-2-1024k policy. And here are the logs from
> datanodes of disconnecting the client:
> datanode 1:
> {noformat}
> 2020-06-15 19:06:20,697 INFO datanode.DataNode: Likely the client has stopped
> reading, disconnecting it (datanode-v11-0-hadoop.hadoop:9866:DataXceiver
> error processing READ_BLOCK operation src: /10.128.23.40:53748 dst:
> /10.128.14.46:9866); java.net.SocketTimeoutException: 1 millis timeout
> while waiting for channel to be ready for write. ch :
> java.nio.channels.SocketChannel[connected local=/10.128.14.46:9866
> remote=/10.128.23.40:53748]
> {noformat}
> datanode 2:
> {noformat}
> 2020-06-15 19:06:20,341 INFO datanode.DataNode: Likely the client has stopped
> reading, disconnecting it (datanode-v11-1-hadoop.hadoop:9866:DataXceiver
> error processing READ_BLOCK operation src
[jira] [Commented] (HDFS-17187) add the tool that getting the hdfs file path through block id
[ https://issues.apache.org/jira/browse/HDFS-17187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17764141#comment-17764141 ] qinyuren commented on HDFS-17187: - *hdfs fsck -blockId * can get the hdfs file path. > add the tool that getting the hdfs file path through block id > - > > Key: HDFS-17187 > URL: https://issues.apache.org/jira/browse/HDFS-17187 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Affects Versions: 3.3.6 >Reporter: liying >Assignee: liying >Priority: Minor > > In some cases, we need to use the block id to get the hdfs file path. > However, the retention time of hdfs namenode log is limited. Maybe we can't > get the block information. If you run fsck / -flies -blocks | grep > '$blockId', you will find that it is very heavy and the hdfs service > performance is affected. So it is need to add the tool that getting the hdfs > file path through block id. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16784) replace readLock with writeLock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage
[ https://issues.apache.org/jira/browse/HDFS-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16784: Summary: replace readLock with writeLock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage (was: use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage) > replace readLock with writeLock in > #DatanodeAdminBackoffMonitor.scanDatanodeStorage > --- > > Key: HDFS-16784 > URL: https://issues.apache.org/jira/browse/HDFS-16784 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: image-2022-09-28-15-37-24-622.png > > > In #DatanodeAdminBackoffMonitor.scanDatanodeStorage, it uses a read lock to > protect the function #isBlockReplicatedOk. > But we found that the function #isBlockReplicatedOk may update the > #neededReconstruction under certain conditions. > !image-2022-09-28-15-37-24-622.png! > Should we replace the read lock with write lock? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16784) use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage
[ https://issues.apache.org/jira/browse/HDFS-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16784: Description: In #DatanodeAdminBackoffMonitor.scanDatanodeStorage, it uses a read lock to protect the function #isBlockReplicatedOk. But we found that the function #isBlockReplicatedOk may update the #neededReconstruction under certain conditions. !image-2022-09-28-15-37-24-622.png! Should we replace the read lock with write lock? was:In > use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage > -- > > Key: HDFS-16784 > URL: https://issues.apache.org/jira/browse/HDFS-16784 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: qinyuren >Priority: Major > Attachments: image-2022-09-28-15-37-24-622.png > > > In #DatanodeAdminBackoffMonitor.scanDatanodeStorage, it uses a read lock to > protect the function #isBlockReplicatedOk. > But we found that the function #isBlockReplicatedOk may update the > #neededReconstruction under certain conditions. > !image-2022-09-28-15-37-24-622.png! > Should we replace the read lock with write lock? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16784) use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage
[ https://issues.apache.org/jira/browse/HDFS-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16784: Attachment: image-2022-09-28-15-37-24-622.png > use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage > -- > > Key: HDFS-16784 > URL: https://issues.apache.org/jira/browse/HDFS-16784 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: qinyuren >Priority: Major > Attachments: image-2022-09-28-15-37-24-622.png > > > In -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16784) use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage
[ https://issues.apache.org/jira/browse/HDFS-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16784: Description: In > use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage > -- > > Key: HDFS-16784 > URL: https://issues.apache.org/jira/browse/HDFS-16784 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: qinyuren >Priority: Major > > In -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16784) use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage
qinyuren created HDFS-16784: --- Summary: use write lock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage Key: HDFS-16784 URL: https://issues.apache.org/jira/browse/HDFS-16784 Project: Hadoop HDFS Issue Type: Bug Reporter: qinyuren -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16575) [SPS]: Should use real replication num instead getReplication from namenode
[
https://issues.apache.org/jira/browse/HDFS-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16575:
Description:
The SPS may have misjudged in the following scenario:
# Create a file with one block and this block have 3 replication with
{color:#de350b}DISK{color} type [DISK, DISK, DISK].
# Set this file with {color:#de350b}ALL_SSD{color} storage policy.
# The replication of this file may become [DISK, DISK,
{color:#de350b}SSD{color}, DISK] with {color:#de350b}decommission{color}.
# Set this file with {color:#de350b}HOT{color} storage policy and satisfy
storage policy on this file.
# The replication finally look like [DISK, DISK, SSD] not [DISK, DISK, DISK]
after decommissioned node offline.
The reason is that SPS get the block replications by
FileStatus.getReplication() which is not the real num of the block.
!image-2022-05-10-11-21-13-627.png|width=432,height=76!
So this block will be ignored, because it have 3 replications with DISK type
already ( one replication in a decommissioning node)
!image-2022-05-10-11-21-31-987.png|width=334,height=179!
I think we can use blockInfo.getLocations().length to count the replication of
block instead of FileStatus.getReplication().
was:
# create a file with one block and this block have 3 replication with
{color:#de350b}DISK{color} type [DISK, DISK, DISK].
# Set this file with {color:#de350b}ALL_SSD{color} storage policy.
# The replication of this file may become [DISK, DISK,
{color:#de350b}SSD{color}, DISK] with {color:#de350b}decommission{color}.
# Set this file with {color:#de350b}HOT{color} storage policy and satisfy
storage policy on this file.
# The replication finally look like [DISK, DISK, SSD] not [DISK, DISK, DISK]
after decommissioned node offline.
The reason is that SPS get the block replications by
FileStatus.getReplication() which is not the real num of the block.
!image-2022-05-10-11-21-13-627.png|width=432,height=76!
So this block will be ignored, because it have 3 replications with DISK type (
one replication in a decommissioning node)
!image-2022-05-10-11-21-31-987.png|width=334,height=179!
> [SPS]: Should use real replication num instead getReplication from namenode
> ---
>
> Key: HDFS-16575
> URL: https://issues.apache.org/jira/browse/HDFS-16575
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Attachments: image-2022-05-10-11-21-13-627.png,
> image-2022-05-10-11-21-31-987.png
>
>
> The SPS may have misjudged in the following scenario:
> # Create a file with one block and this block have 3 replication with
> {color:#de350b}DISK{color} type [DISK, DISK, DISK].
> # Set this file with {color:#de350b}ALL_SSD{color} storage policy.
> # The replication of this file may become [DISK, DISK,
> {color:#de350b}SSD{color}, DISK] with {color:#de350b}decommission{color}.
> # Set this file with {color:#de350b}HOT{color} storage policy and satisfy
> storage policy on this file.
> # The replication finally look like [DISK, DISK, SSD] not [DISK, DISK,
> DISK] after decommissioned node offline.
> The reason is that SPS get the block replications by
> FileStatus.getReplication() which is not the real num of the block.
> !image-2022-05-10-11-21-13-627.png|width=432,height=76!
> So this block will be ignored, because it have 3 replications with DISK type
> already ( one replication in a decommissioning node)
> !image-2022-05-10-11-21-31-987.png|width=334,height=179!
> I think we can use blockInfo.getLocations().length to count the replication
> of block instead of FileStatus.getReplication().
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16575) [SPS]: Should use real replication num instead getReplication from namenode
[
https://issues.apache.org/jira/browse/HDFS-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16575:
Description:
# create a file with one block and this block have 3 replication with
{color:#de350b}DISK{color} type [DISK, DISK, DISK].
# Set this file with {color:#de350b}ALL_SSD{color} storage policy.
# The replication of this file may become [DISK, DISK,
{color:#de350b}SSD{color}, DISK] with {color:#de350b}decommission{color}.
# Set this file with {color:#de350b}HOT{color} storage policy and satisfy
storage policy on this file.
# The replication finally look like [DISK, DISK, SSD] not [DISK, DISK, DISK]
after decommissioned node offline.
The reason is that SPS get the block replications by
FileStatus.getReplication() which is not the real num of the block.
!image-2022-05-10-11-21-13-627.png|width=432,height=76!
So this block will be ignored, because it have 3 replications with DISK type (
one replication in a decommissioning node)
!image-2022-05-10-11-21-31-987.png|width=334,height=179!
> [SPS]: Should use real replication num instead getReplication from namenode
> ---
>
> Key: HDFS-16575
> URL: https://issues.apache.org/jira/browse/HDFS-16575
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Attachments: image-2022-05-10-11-21-13-627.png,
> image-2022-05-10-11-21-31-987.png
>
>
> # create a file with one block and this block have 3 replication with
> {color:#de350b}DISK{color} type [DISK, DISK, DISK].
> # Set this file with {color:#de350b}ALL_SSD{color} storage policy.
> # The replication of this file may become [DISK, DISK,
> {color:#de350b}SSD{color}, DISK] with {color:#de350b}decommission{color}.
> # Set this file with {color:#de350b}HOT{color} storage policy and satisfy
> storage policy on this file.
> # The replication finally look like [DISK, DISK, SSD] not [DISK, DISK,
> DISK] after decommissioned node offline.
> The reason is that SPS get the block replications by
> FileStatus.getReplication() which is not the real num of the block.
> !image-2022-05-10-11-21-13-627.png|width=432,height=76!
> So this block will be ignored, because it have 3 replications with DISK type
> ( one replication in a decommissioning node)
> !image-2022-05-10-11-21-31-987.png|width=334,height=179!
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16575) [SPS]: Should use real replication num instead getReplication from namenode
[ https://issues.apache.org/jira/browse/HDFS-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16575: Attachment: image-2022-05-10-11-21-31-987.png > [SPS]: Should use real replication num instead getReplication from namenode > --- > > Key: HDFS-16575 > URL: https://issues.apache.org/jira/browse/HDFS-16575 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: qinyuren >Priority: Major > Attachments: image-2022-05-10-11-21-13-627.png, > image-2022-05-10-11-21-31-987.png > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16575) [SPS]: Should use real replication num instead getReplication from namenode
[ https://issues.apache.org/jira/browse/HDFS-16575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16575: Attachment: image-2022-05-10-11-21-13-627.png > [SPS]: Should use real replication num instead getReplication from namenode > --- > > Key: HDFS-16575 > URL: https://issues.apache.org/jira/browse/HDFS-16575 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: qinyuren >Priority: Major > Attachments: image-2022-05-10-11-21-13-627.png, > image-2022-05-10-11-21-31-987.png > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16575) [SPS]: Should use real replication num instead getReplication from namenode
qinyuren created HDFS-16575: --- Summary: [SPS]: Should use real replication num instead getReplication from namenode Key: HDFS-16575 URL: https://issues.apache.org/jira/browse/HDFS-16575 Project: Hadoop HDFS Issue Type: Bug Reporter: qinyuren -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer
[
https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16544:
Description:
In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we
found an EC file decoding bug if more than one data block read failed.
Currently, we found another bug trigger by #StatefulStripeReader.decode.
If we read an EC file which {*}length more than one stripe{*}, and this file
have *one data block* and *the first parity block* corrupted, this error will
happen.
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
Let's say we use ec(6+3) and the data block[0] and the first parity block[6]
are corrupted.
# The readers for block[0] and block[6] will be closed after reading the first
stripe of an EC file;
# When the client reading the second stripe of the EC file, it will trigger
#prepareParityChunk for block[6].
# The decodeInputs[6] will not be constructed because the reader for block[6]
was closed.
{code:java}
boolean prepareParityChunk(int index) {
Preconditions.checkState(index >= dataBlkNum
&& alignedStripe.chunks[index] == null);
if (readerInfos[index] != null && readerInfos[index].shouldSkip) {
alignedStripe.chunks[index] = new StripingChunk(StripingChunk.MISSING);
// we have failed the block reader before
return false;
}
final int parityIndex = index - dataBlkNum;
ByteBuffer buf = dfsStripedInputStream.getParityBuffer().duplicate();
buf.position(cellSize * parityIndex);
buf.limit(cellSize * parityIndex + (int) alignedStripe.range.spanInBlock);
decodeInputs[index] =
new ECChunk(buf.slice(), 0, (int) alignedStripe.range.spanInBlock);
alignedStripe.chunks[index] =
new StripingChunk(decodeInputs[index].getBuffer());
return true;
} {code}
was:
In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we
found an EC file decoding bug if more than one data block read failed.
Currently, we found another bug trigger by #StatefulStripeReader.decode.
If we read an EC file which {*}length more than one stripe{*}, and this file
have *one data block* and *the first parity block* corrupted, this error will
happen.
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
Let's say we use ec(6+3) and the data block[0] and the first parity block[6]
are corrupted.
# The readers for block[0] and block[6] will be closed after reading the first
stripe of an EC file;
# When the client reading the second stripe of the EC file, it will trigger
#prepareParityChunk for block[6].
# The decodeInputs[6] will not be constructed due to the reader for block[6]
was closed.
{code:java}
boolean prepareParityChunk(int index) {
Preconditions.checkState(index >= dataBlkNum
&& alignedStripe.chunks[index] == null);
if (readerInfos[index] != null && readerInfos[index].shouldSkip) {
alignedStr
[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer
[
https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16544:
Description:
In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we
found an EC file decoding bug if more than one data block read failed.
Currently, we found another bug trigger by #StatefulStripeReader.decode.
If we read an EC file which {*}length more than one stripe{*}, and this file
have *one data block* and *the first parity block* corrupted, this error will
happen.
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
Let's say we use ec(6+3) and the data block[0] and the first parity block[6]
are corrupted.
# The readers for block[0] and block[6] will be closed after reading the first
stripe of an EC file;
# When the client reading the second stripe of the EC file, it will trigger
#prepareParityChunk for block[6].
# The decodeInputs[6] will not be constructed due to the reader for block[6]
was closed.
{code:java}
boolean prepareParityChunk(int index) {
Preconditions.checkState(index >= dataBlkNum
&& alignedStripe.chunks[index] == null);
if (readerInfos[index] != null && readerInfos[index].shouldSkip) {
alignedStripe.chunks[index] = new StripingChunk(StripingChunk.MISSING);
// we have failed the block reader before
return false;
}
final int parityIndex = index - dataBlkNum;
ByteBuffer buf = dfsStripedInputStream.getParityBuffer().duplicate();
buf.position(cellSize * parityIndex);
buf.limit(cellSize * parityIndex + (int) alignedStripe.range.spanInBlock);
decodeInputs[index] =
new ECChunk(buf.slice(), 0, (int) alignedStripe.range.spanInBlock);
alignedStripe.chunks[index] =
new StripingChunk(decodeInputs[index].getBuffer());
return true;
} {code}
was:
In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we
found an EC file decoding bug if more than one data block read failed.
Currently, we found another bug trigger by #StatefulStripeReader.decode.
If we read an EC file which {*}length more than one stripe{*}, and this file
have *one data block* and *the first parity block* corrupted, this error will
happen.
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
> EC decoding failed due to invalid buffer
>
>
> Key: HDFS-16544
> URL: https://issues.apache.org/jira/browse/HDFS-16544
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: qinyuren
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we
> found an EC file decoding bug if more than one data block read failed.
> Currently, we found ano
[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer
[
https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16544:
Description:
In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we
found an EC file decoding bug if more than one data block read failed.
Currently, we found another bug trigger by #StatefulStripeReader.decode.
If we read an EC file which {*}length more than one stripe{*}, and this file
have *one data block* and *the first parity block* corrupted, this error will
happen.
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
was:
In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we
found a ec decoding bug if more than one data block read failed.
Currently, we found another bug trigger by #StatefulStripeReader.decode.
If we read an EC file which {*}length more than one stripe{*}, and this file
have *one data block* and *the first parity block* corrupted, this error will
happen.
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
> EC decoding failed due to invalid buffer
>
>
> Key: HDFS-16544
> URL: https://issues.apache.org/jira/browse/HDFS-16544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
>
> In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we
> found an EC file decoding bug if more than one data block read failed.
> Currently, we found another bug trigger by #StatefulStripeReader.decode.
> If we read an EC file which {*}length more than one stripe{*}, and this file
> have *one data block* and *the first parity block* corrupted, this error will
> happen.
> {code:java}
> org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
> allowing null at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
> {code}
--
This mess
[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer
[
https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16544:
Description:
In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we
found a ec decoding bug if more than one data block read failed.
Currently, we found another bug trigger by #StatefulStripeReader.decode.
If we read an EC file which {*}length more than one stripe{*}, and this file
have *one data block* and *the first parity block* corrupted, this error will
happen.
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
was:
In [HDFS-16538|http://https://issues.apache.org/jira/browse/HDFS-16538] , we
found a ec decoding bug if more than one data block read failed.
Currently, we found another bug trigger by #StatefulStripeReader.decode.
If we read an EC file which length more than one stripe, and this file have one
data block and the first parity block corrupted, this error will happen.
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
> EC decoding failed due to invalid buffer
>
>
> Key: HDFS-16544
> URL: https://issues.apache.org/jira/browse/HDFS-16544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
>
> In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538] , we
> found a ec decoding bug if more than one data block read failed.
> Currently, we found another bug trigger by #StatefulStripeReader.decode.
> If we read an EC file which {*}length more than one stripe{*}, and this file
> have *one data block* and *the first parity block* corrupted, this error will
> happen.
> {code:java}
> org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
> allowing null at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
> {code}
--
This message was sent by Atlass
[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer
[
https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16544:
Description:
In [HDFS-16538|http://https://issues.apache.org/jira/browse/HDFS-16538] , we
found a ec decoding bug if more than one data block read failed.
Currently, we found another bug trigger by #StatefulStripeReader.decode.
If we read an EC file which length more than one stripe, and this file have one
data block and the first parity block corrupted, this error will happen.
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
was:
In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538]
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
> EC decoding failed due to invalid buffer
>
>
> Key: HDFS-16544
> URL: https://issues.apache.org/jira/browse/HDFS-16544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
>
> In [HDFS-16538|http://https://issues.apache.org/jira/browse/HDFS-16538] , we
> found a ec decoding bug if more than one data block read failed.
> Currently, we found another bug trigger by #StatefulStripeReader.decode.
> If we read an EC file which length more than one stripe, and this file have
> one data block and the first parity block corrupted, this error will happen.
> {code:java}
> org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
> allowing null at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer
[
https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16544:
Description:
In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538]
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
was:
In [HDFS-16538|http://https://issues.apache.org/jira/browse/HDFS-16538]
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
> EC decoding failed due to invalid buffer
>
>
> Key: HDFS-16544
> URL: https://issues.apache.org/jira/browse/HDFS-16544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
>
> In [HDFS-16538|http://https//issues.apache.org/jira/browse/HDFS-16538]
>
> {code:java}
> org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
> allowing null at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16544) EC decoding failed due to invalid buffer
[
https://issues.apache.org/jira/browse/HDFS-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16544:
Description:
In [HDFS-16538|http://https://issues.apache.org/jira/browse/HDFS-16538]
{code:java}
org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
allowing null at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
{code}
> EC decoding failed due to invalid buffer
>
>
> Key: HDFS-16544
> URL: https://issues.apache.org/jira/browse/HDFS-16544
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
>
> In [HDFS-16538|http://https://issues.apache.org/jira/browse/HDFS-16538]
>
> {code:java}
> org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer found, not
> allowing null at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers(ByteBufferDecodingState.java:132)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:48)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:435)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:392)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:315)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:408)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:918)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16544) EC decoding failed due to invalid buffer
qinyuren created HDFS-16544: --- Summary: EC decoding failed due to invalid buffer Key: HDFS-16544 URL: https://issues.apache.org/jira/browse/HDFS-16544 Project: Hadoop HDFS Issue Type: Bug Reporter: qinyuren -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16538:
Description:
Currently, we found this error if the #StripeReader.readStripe() have more than
one block read failed.
We use the EC policy ec(6+3) in our cluster.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
} {code}
This error can be trigger by #StatefulStripeReader.decode.
The reason is that:
# If there are more than one *data block* read failed, the
#readDataForDecoding will be called multiple times;
# The *decodeInputs array* will be initialized repeatedly.
# The *parity* *data* in *decodeInputs array* which filled by
#readParityChunks previously will be set to null.
was:
Currently, we found this error if the #StripeReader.readStripe() have more than
one block read failed.
We use the EC policy ec(6+3) in our cluster.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16538:
Description:
Currently, we found this error if the #StripeReader.readStripe() have more than
one block read failed.
We use the EC policy ec(6+3) in our cluster.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
} {code}
This error can be trigger by #StatefulStripeReader.decode.
The reason is that:
1. If there are more than one *data block* read failed, the
#readDataForDecoding will be called multiple times;
The *decodeInputs array* will be initialized repeatedly, and the *parity*
*data* in *decodeInputs array* which filled by #readParityChunks will be set to
null.
was:
Currently, we found this error if the #StripeReader.readStripe() have more than
one block read failed.
We use the EC policy ec(6+3) in our cluster.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk return
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16538:
Description:
Currently, we found this error if the #StripeReader.readStripe() have more than
one block read failed.
We use the EC policy ec(6+3) in our cluster.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
} {code}
If there are two blocks read failed, the #readDataForDecoding() will be called
twice;
The *decodeInputs array* will be initialized twice, and the *parity* *data* in
decodeInputs array which filled by #readParityChunks will be set to null.
was:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16538:
Description:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
} {code}
If there are two blocks read failed, the #readDataForDecoding() will be called
twice;
The *decodeInputs array* will be initialized twice, and the *parity* *data* in
decodeInputs array which filled by #readParityChunks will be set to null.
was:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL)
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16538:
Description:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
} {code}
If there are two blocks read failed, the #readDataForDecoding() will be called
twice;
The decodeInputs[] will be initialized twice, the *parity* data in
decodeInputs which filled by #readParityChunks will be set to null.
was:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedC
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16538:
Description:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
} {code}
If there are two blocks read failed, the #readDataForDecoding() will be called
twice;
The decodeInputs[] will be initialized twice, this cause the *parity* data in
decodeInputs which fill by #readParityChunks miss.
> EC decoding failed due to not enough valid inputs
> --
>
> Key: HDFS-16538
> URL: https://issues.apache.org/jira/browse/HDFS-16538
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
>
> We found this error if the #StripeReader.readStripe() have more than one
> block read failed.
>
> {code:java}
> Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
> inputs are provided, not recoverable
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at
> org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
> at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
> at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
> {code}
>
>
> {code:java}
> while (!futures.isEmpty()) {
> try {
> StripingChunkReadResult r = StripedBlockUtil
> .getNextCompletedStripedRead(service, futures, 0);
> dfsStripedInpu
[jira] [Created] (HDFS-16538) EC decoding failed due to not enough valid inputs
qinyuren created HDFS-16538: --- Summary: EC decoding failed due to not enough valid inputs Key: HDFS-16538 URL: https://issues.apache.org/jira/browse/HDFS-16538 Project: Hadoop HDFS Issue Type: Bug Reporter: qinyuren -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread
[
https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16484:
Description:
Currently, we ran SPS in our cluster and found this log. The SPSPathIdProcessor
thread enters an infinite loop and prints the same log all the time.
!image-2022-02-25-14-35-42-255.png|width=682,height=195!
In SPSPathIdProcessor thread, if it get a inodeId which path does not exist,
then the SPSPathIdProcessor thread entry infinite loop and can't work normally.
The reason is that #ctxt.getNextSPSPath() get a inodeId which path does not
exist. The inodeId will not be set to null, causing the thread hold this
inodeId forever.
{code:java}
public void run() {
LOG.info("Starting SPSPathIdProcessor!.");
Long startINode = null;
while (ctxt.isRunning()) {
try {
if (!ctxt.isInSafeMode()) {
if (startINode == null) {
startINode = ctxt.getNextSPSPath();
} // else same id will be retried
if (startINode == null) {
// Waiting for SPS path
Thread.sleep(3000);
} else {
ctxt.scanAndCollectFiles(startINode);
// check if directory was empty and no child added to queue
DirPendingWorkInfo dirPendingWorkInfo =
pendingWorkForDirectory.get(startINode);
if (dirPendingWorkInfo != null
&& dirPendingWorkInfo.isDirWorkDone()) {
ctxt.removeSPSHint(startINode);
pendingWorkForDirectory.remove(startINode);
}
}
startINode = null; // Current inode successfully scanned.
}
} catch (Throwable t) {
String reClass = t.getClass().getName();
if (InterruptedException.class.getName().equals(reClass)) {
LOG.info("SPSPathIdProcessor thread is interrupted. Stopping..");
break;
}
LOG.warn("Exception while scanning file inodes to satisfy the policy",
t);
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
LOG.info("Interrupted while waiting in SPSPathIdProcessor", t);
break;
}
}
}
} {code}
was:
In SPSPathIdProcessor thread, if it get a inodeId which path does not exist,
then the SPSPathIdProcessor thread entry infinite loop and can't work normally.
!image-2022-02-25-14-35-42-255.png|width=682,height=195!
> [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread
> -
>
> Key: HDFS-16484
> URL: https://issues.apache.org/jira/browse/HDFS-16484
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: qinyuren
>Assignee: qinyuren
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-02-25-14-35-42-255.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently, we ran SPS in our cluster and found this log. The
> SPSPathIdProcessor thread enters an infinite loop and prints the same log all
> the time.
> !image-2022-02-25-14-35-42-255.png|width=682,height=195!
> In SPSPathIdProcessor thread, if it get a inodeId which path does not exist,
> then the SPSPathIdProcessor thread entry infinite loop and can't work
> normally.
> The reason is that #ctxt.getNextSPSPath() get a inodeId which path does not
> exist. The inodeId will not be set to null, causing the thread hold this
> inodeId forever.
> {code:java}
> public void run() {
> LOG.info("Starting SPSPathIdProcessor!.");
> Long startINode = null;
> while (ctxt.isRunning()) {
> try {
> if (!ctxt.isInSafeMode()) {
> if (startINode == null) {
> startINode = ctxt.getNextSPSPath();
> } // else same id will be retried
> if (startINode == null) {
> // Waiting for SPS path
> Thread.sleep(3000);
> } else {
> ctxt.scanAndCollectFiles(startINode);
> // check if directory was empty and no child added to queue
> DirPendingWorkInfo dirPendingWorkInfo =
> pendingWorkForDirectory.get(startINode);
> if (dirPendingWorkInfo != null
> && dirPendingWorkInfo.isDirWorkDone()) {
> ctxt.removeSPSHint(startINode);
> pendingWorkForDirectory.remove(startINode);
> }
> }
> startINode = null; // Current inode successfully scanned.
> }
> } catch (Throwable t) {
> String reClass = t.getClass().getName();
> if (InterruptedException.class.getName().equals(reClass)) {
> LOG.info("SPSPathIdProcessor thread is interrupted. Stopping..");
> break;
> }
> LOG.warn("Exception while scanning file inodes to satisfy the policy",
> t);
> try {
> Thread.sleep(3000);
> } catch (InterruptedException e) {
> LOG.info("Interrupted while waiting in
[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode as follow: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! I think we can reduce the sleep time of the first several failover based on the number of namenode For example, if 4 namenode are configured, the sleep time of first three failover operations is set to zero. was: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode as follow: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! I think we can reduce the sleep time of the first several failover based on the number of namenode For example, if 4 namenode are configured, the sleep time of first three failover operations is zero. > Reduce the failover sleep time if multiple namenode are configured > -- > > Key: HDFS-16514 > URL: https://issues.apache.org/jira/browse/HDFS-16514 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2022-03-21-18-11-37-191.png > > > Recently, we used the [Standby Read] feature in our test cluster, and > deployed 4 namenode as follow: > node1 -> active nn > node2 -> standby nn > node3 -> observer nn > node3 -> observer nn > If we set ’dfs.client.failover.random.order=true‘, the client may failover > twice and wait a long time to send msync to active namenode. > !image-2022-03-21-18-11-37-191.png|width=698,height=169! > I think we can reduce the sleep time of the first several failover based on > the number of namenode > For example, if 4 namenode are configured, the sleep time of first three > failover operations is set to zero. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode as follow: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! I think we can reduce the sleep time of the first several failover based on the number of namenode For example, if 4 namenode are configured, the sleep time of first three failover operations is zero. was: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode as follow: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! I think we can reduce the sleep time of the first several failover based on the number of namenode > Reduce the failover sleep time if multiple namenode are configured > -- > > Key: HDFS-16514 > URL: https://issues.apache.org/jira/browse/HDFS-16514 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2022-03-21-18-11-37-191.png > > > Recently, we used the [Standby Read] feature in our test cluster, and > deployed 4 namenode as follow: > node1 -> active nn > node2 -> standby nn > node3 -> observer nn > node3 -> observer nn > If we set ’dfs.client.failover.random.order=true‘, the client may failover > twice and wait a long time to send msync to active namenode. > !image-2022-03-21-18-11-37-191.png|width=698,height=169! > I think we can reduce the sleep time of the first several failover based on > the number of namenode > For example, if 4 namenode are configured, the sleep time of first three > failover operations is zero. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode as follow: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! I think we can reduce the sleep time of the first several failover based on the number of namenode was: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! I think we can reduce the sleep time of the first several failover based on the number of namenode > Reduce the failover sleep time if multiple namenode are configured > -- > > Key: HDFS-16514 > URL: https://issues.apache.org/jira/browse/HDFS-16514 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2022-03-21-18-11-37-191.png > > > Recently, we used the [Standby Read] feature in our test cluster, and > deployed 4 namenode as follow: > node1 -> active nn > node2 -> standby nn > node3 -> observer nn > node3 -> observer nn > If we set ’dfs.client.failover.random.order=true‘, the client may failover > twice and wait a long time to send msync to active namenode. > !image-2022-03-21-18-11-37-191.png|width=698,height=169! > I think we can reduce the sleep time of the first several failover based on > the number of namenode -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! I think we can reduce the sleep time of the first several failover based on the number of namenode was: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! Perhaps we can reduce the sleep time of the first several failover based on the number of namenode > Reduce the failover sleep time if multiple namenode are configured > -- > > Key: HDFS-16514 > URL: https://issues.apache.org/jira/browse/HDFS-16514 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2022-03-21-18-11-37-191.png > > > Recently, we used the [Standby Read] feature in our test cluster, and > deployed 4 namenode: > node1 -> active nn > node2 -> standby nn > node3 -> observer nn > node3 -> observer nn > If we set ’dfs.client.failover.random.order=true‘, the client may failover > twice and wait a long time to send msync to active namenode. > !image-2022-03-21-18-11-37-191.png|width=698,height=169! > I think we can reduce the sleep time of the first several failover based on > the number of namenode -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! Perhaps we can reduce the sleep time of the first several failover based on the number of namenode was: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! > Reduce the failover sleep time if multiple namenode are configured > -- > > Key: HDFS-16514 > URL: https://issues.apache.org/jira/browse/HDFS-16514 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2022-03-21-18-11-37-191.png > > > Recently, we used the [Standby Read] feature in our test cluster, and > deployed 4 namenode: > node1 -> active nn > node2 -> standby nn > node3 -> observer nn > node3 -> observer nn > If we set ’dfs.client.failover.random.order=true‘, the client may failover > twice and wait a long time to send msync to active namenode. > !image-2022-03-21-18-11-37-191.png|width=698,height=169! > Perhaps we can reduce the sleep time of the first several failover based on > the number of namenode -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=661,height=160! was: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=987,height=239! > Reduce the failover sleep time if multiple namenode are configured > -- > > Key: HDFS-16514 > URL: https://issues.apache.org/jira/browse/HDFS-16514 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2022-03-21-18-11-37-191.png > > > Recently, we used the [Standby Read] feature in our test cluster, and > deployed 4 namenode: > node1 -> active nn > node2 -> standby nn > node3 -> observer nn > node3 -> observer nn > If we set ’dfs.client.failover.random.order=true‘, the client may failover > twice and wait a long time to send msync to active namenode. > !image-2022-03-21-18-11-37-191.png|width=661,height=160! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured
[ https://issues.apache.org/jira/browse/HDFS-16514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16514: Description: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=698,height=169! was: Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=661,height=160! > Reduce the failover sleep time if multiple namenode are configured > -- > > Key: HDFS-16514 > URL: https://issues.apache.org/jira/browse/HDFS-16514 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2022-03-21-18-11-37-191.png > > > Recently, we used the [Standby Read] feature in our test cluster, and > deployed 4 namenode: > node1 -> active nn > node2 -> standby nn > node3 -> observer nn > node3 -> observer nn > If we set ’dfs.client.failover.random.order=true‘, the client may failover > twice and wait a long time to send msync to active namenode. > !image-2022-03-21-18-11-37-191.png|width=698,height=169! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured
qinyuren created HDFS-16514: --- Summary: Reduce the failover sleep time if multiple namenode are configured Key: HDFS-16514 URL: https://issues.apache.org/jira/browse/HDFS-16514 Project: Hadoop HDFS Issue Type: Improvement Reporter: qinyuren Attachments: image-2022-03-21-18-11-37-191.png Recently, we used the [Standby Read] feature in our test cluster, and deployed 4 namenode: node1 -> active nn node2 -> standby nn node3 -> observer nn node3 -> observer nn If we set ’dfs.client.failover.random.order=true‘, the client may failover twice and wait a long time to send msync to active namenode. !image-2022-03-21-18-11-37-191.png|width=987,height=239! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16501) Print the exception when reporting a bad block
qinyuren created HDFS-16501: --- Summary: Print the exception when reporting a bad block Key: HDFS-16501 URL: https://issues.apache.org/jira/browse/HDFS-16501 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Reporter: qinyuren Attachments: image-2022-03-10-19-27-31-622.png !image-2022-03-10-19-27-31-622.png|width=847,height=27! Currently, volumeScanner will find bad block and report it to namenode without printing the reason why the block is a bad block. I think we should be better print the exception in log file. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16485) [SPS]: allow re-satisfy path after restarting sps process
[
https://issues.apache.org/jira/browse/HDFS-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16485:
Description:
When SPSPathIdProcessor thread call getNextSPSPath(), it get the pathId from
namenode and namenode will also remove this pathId from pathsToBeTraveresed
queue.
{code:java}
public Long getNextPathId() {
synchronized (pathsToBeTraveresed) {
return pathsToBeTraveresed.poll();
}
} {code}
If SPS process restart, this path will not continue the move operation until
namenode restart.
So we want to provide a way for the SPS to continue performing the move
operation after SPS restart.
First solution:
1) When SPSPathIdProcessor thread call getNextSPSPath(), namenode return pathId
and then move this pathId to a pathsBeingTraveresed queue;
2) After SPS finish a path movement operation, it call a rpc to namenode to
remove this pathId from pathsBeingTraveresed queue;
3) If SPS restart, SPSPathIdProcessor thread should call a rpc to namenode to
get all pathId from pathsBeingTraveresed queue;
Second solution:
We added timeout detection in the application layer, if a path does not
complete the movement within the specified time, we can re-satisfy this path
even though it has "hdfs.sps" xattr already.
We choose the second solution because the first solution will add more rpc
operation and may affect namenode performance.
was:
When SPSPathIdProcessor thread call getNextSPSPath(), it get the pathId from
namenode and namenode will also remove this pathId from pathsToBeTraveresed
queue.
{code:java}
public Long getNextPathId() {
synchronized (pathsToBeTraveresed) {
return pathsToBeTraveresed.poll();
}
} {code}
If SPS process restart, this path will not continue the move operation until
namenode restart.
So we want to provide a way for the SPS to continue performing the move
operation after SPS restart.
First solution:
1) When SPSPathIdProcessor thread call getNextSPSPath(), namenode return pathId
and then move this pathId to a pathsBeingTraveresed queue;
2) After SPS finish a path movement operation, it call a rpc to namenode to
remove this pathId from pathsBeingTraveresed queue;
3) If SPS restart, SPSPathIdProcessor thread should call a rpc to namenode to
get all pathId from pathsBeingTraveresed queue;
Second solution:
We added timeout detection in the application layer, if a path does not
complete the movement within the specified time, we can re-satisfy this path
even though it has "hdfs.sps" xattr already.
We choose the second solution because the first solution will add more rpc
operation
> [SPS]: allow re-satisfy path after restarting sps process
> -
>
> Key: HDFS-16485
> URL: https://issues.apache.org/jira/browse/HDFS-16485
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: qinyuren
>Priority: Major
>
> When SPSPathIdProcessor thread call getNextSPSPath(), it get the pathId from
> namenode and namenode will also remove this pathId from pathsToBeTraveresed
> queue.
> {code:java}
> public Long getNextPathId() {
> synchronized (pathsToBeTraveresed) {
> return pathsToBeTraveresed.poll();
> }
> } {code}
> If SPS process restart, this path will not continue the move operation until
> namenode restart.
> So we want to provide a way for the SPS to continue performing the move
> operation after SPS restart.
> First solution:
> 1) When SPSPathIdProcessor thread call getNextSPSPath(), namenode return
> pathId and then move this pathId to a pathsBeingTraveresed queue;
> 2) After SPS finish a path movement operation, it call a rpc to namenode to
> remove this pathId from pathsBeingTraveresed queue;
> 3) If SPS restart, SPSPathIdProcessor thread should call a rpc to namenode to
> get all pathId from pathsBeingTraveresed queue;
> Second solution:
> We added timeout detection in the application layer, if a path does not
> complete the movement within the specified time, we can re-satisfy this path
> even though it has "hdfs.sps" xattr already.
> We choose the second solution because the first solution will add more rpc
> operation and may affect namenode performance.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16485) [SPS]: allow re-satisfy path after restarting sps process
[
https://issues.apache.org/jira/browse/HDFS-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16485:
Description:
When SPSPathIdProcessor thread call getNextSPSPath(), it get the pathId from
namenode and namenode will also remove this pathId from pathsToBeTraveresed
queue.
{code:java}
public Long getNextPathId() {
synchronized (pathsToBeTraveresed) {
return pathsToBeTraveresed.poll();
}
} {code}
If SPS process restart, this path will not continue the move operation until
namenode restart.
So we want to provide a way for the SPS to continue performing the move
operation after SPS restart.
First solution:
1) When SPSPathIdProcessor thread call getNextSPSPath(), namenode return pathId
and then move this pathId to a pathsBeingTraveresed queue;
2) After SPS finish a path movement operation, it call a rpc to namenode to
remove this pathId from pathsBeingTraveresed queue;
3) If SPS restart, SPSPathIdProcessor thread should call a rpc to namenode to
get all pathId from pathsBeingTraveresed queue;
Second solution:
We added timeout detection in the application layer, if a path does not
complete the movement within the specified time, we can re-satisfy this path
even though it has "hdfs.sps" xattr already.
We choose the second solution because the first solution will add more rpc
operation
was:
{code:java}
public Long getNextPathId() {
synchronized (pathsToBeTraveresed) {
return pathsToBeTraveresed.poll();
}
} {code}
> [SPS]: allow re-satisfy path after restarting sps process
> -
>
> Key: HDFS-16485
> URL: https://issues.apache.org/jira/browse/HDFS-16485
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: qinyuren
>Priority: Major
>
> When SPSPathIdProcessor thread call getNextSPSPath(), it get the pathId from
> namenode and namenode will also remove this pathId from pathsToBeTraveresed
> queue.
> {code:java}
> public Long getNextPathId() {
> synchronized (pathsToBeTraveresed) {
> return pathsToBeTraveresed.poll();
> }
> } {code}
> If SPS process restart, this path will not continue the move operation until
> namenode restart.
> So we want to provide a way for the SPS to continue performing the move
> operation after SPS restart.
> First solution:
> 1) When SPSPathIdProcessor thread call getNextSPSPath(), namenode return
> pathId and then move this pathId to a pathsBeingTraveresed queue;
> 2) After SPS finish a path movement operation, it call a rpc to namenode to
> remove this pathId from pathsBeingTraveresed queue;
> 3) If SPS restart, SPSPathIdProcessor thread should call a rpc to namenode to
> get all pathId from pathsBeingTraveresed queue;
> Second solution:
> We added timeout detection in the application layer, if a path does not
> complete the movement within the specified time, we can re-satisfy this path
> even though it has "hdfs.sps" xattr already.
> We choose the second solution because the first solution will add more rpc
> operation
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16485) [SPS]: allow re-satisfy path after restarting sps process
[
https://issues.apache.org/jira/browse/HDFS-16485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16485:
Description:
{code:java}
public Long getNextPathId() {
synchronized (pathsToBeTraveresed) {
return pathsToBeTraveresed.poll();
}
} {code}
> [SPS]: allow re-satisfy path after restarting sps process
> -
>
> Key: HDFS-16485
> URL: https://issues.apache.org/jira/browse/HDFS-16485
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: qinyuren
>Priority: Major
>
>
> {code:java}
> public Long getNextPathId() {
> synchronized (pathsToBeTraveresed) {
> return pathsToBeTraveresed.poll();
> }
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16485) [SPS]: allow re-satisfy path after restarting sps process
qinyuren created HDFS-16485: --- Summary: [SPS]: allow re-satisfy path after restarting sps process Key: HDFS-16485 URL: https://issues.apache.org/jira/browse/HDFS-16485 Project: Hadoop HDFS Issue Type: Sub-task Reporter: qinyuren -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren reassigned HDFS-16484: --- Assignee: qinyuren > [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread > - > > Key: HDFS-16484 > URL: https://issues.apache.org/jira/browse/HDFS-16484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: qinyuren >Assignee: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: image-2022-02-25-14-35-42-255.png > > Time Spent: 10m > Remaining Estimate: 0h > > In SPSPathIdProcessor thread, if it get a inodeId which path does not exist, > then the SPSPathIdProcessor thread entry infinite loop and can't work > normally. > !image-2022-02-25-14-35-42-255.png|width=682,height=195! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Description: In SPSPathIdProcessor thread, if it get a inodeId which path does not exist, then the SPSPathIdProcessor thread entry infinite loop and can't work normally. !image-2022-02-25-14-35-42-255.png|width=682,height=195! was: In SPSPathIdProcessor thread, if sps get a inodeId which path does not exist, then the SPSPathIdProcessor thread entry an !image-2022-02-25-14-35-42-255.png|width=682,height=195! > [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread > - > > Key: HDFS-16484 > URL: https://issues.apache.org/jira/browse/HDFS-16484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: qinyuren >Priority: Major > Attachments: image-2022-02-25-14-35-42-255.png > > > In SPSPathIdProcessor thread, if it get a inodeId which path does not exist, > then the SPSPathIdProcessor thread entry infinite loop and can't work > normally. > !image-2022-02-25-14-35-42-255.png|width=682,height=195! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Description: In SPSPathIdProcessor thread, if sps get a inodeId which path does not exist, then the SPSPathIdProcessor thread entry an !image-2022-02-25-14-35-42-255.png|width=682,height=195! was:!image-2022-02-25-14-35-42-255.png|width=682,height=195! > [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread > - > > Key: HDFS-16484 > URL: https://issues.apache.org/jira/browse/HDFS-16484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: qinyuren >Priority: Major > Attachments: image-2022-02-25-14-35-42-255.png > > > In SPSPathIdProcessor thread, if sps get a inodeId which path does not exist, > then the SPSPathIdProcessor thread entry an > !image-2022-02-25-14-35-42-255.png|width=682,height=195! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Description: !image-2022-02-25-14-35-42-255.png|width=682,height=195! > [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread > - > > Key: HDFS-16484 > URL: https://issues.apache.org/jira/browse/HDFS-16484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: qinyuren >Priority: Major > Attachments: image-2022-02-25-14-35-42-255.png > > > !image-2022-02-25-14-35-42-255.png|width=682,height=195! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Attachment: image-2022-02-25-14-35-42-255.png > [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread > - > > Key: HDFS-16484 > URL: https://issues.apache.org/jira/browse/HDFS-16484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: qinyuren >Priority: Major > Attachments: image-2022-02-25-14-35-42-255.png > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16484) [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Summary: [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (was: [SPS]: fix an infinite loop bug in SPSPathIdProcessor thread ) > [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread > - > > Key: HDFS-16484 > URL: https://issues.apache.org/jira/browse/HDFS-16484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: qinyuren >Priority: Major > Attachments: image-2022-02-25-14-35-42-255.png > > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16484) fix an infinite loop bug in SPSPathIdProcessor thread
qinyuren created HDFS-16484: --- Summary: fix an infinite loop bug in SPSPathIdProcessor thread Key: HDFS-16484 URL: https://issues.apache.org/jira/browse/HDFS-16484 Project: Hadoop HDFS Issue Type: Sub-task Reporter: qinyuren -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16484) [SPS]: fix an infinite loop bug in SPSPathIdProcessor thread
[ https://issues.apache.org/jira/browse/HDFS-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16484: Summary: [SPS]: fix an infinite loop bug in SPSPathIdProcessor thread (was: fix an infinite loop bug in SPSPathIdProcessor thread ) > [SPS]: fix an infinite loop bug in SPSPathIdProcessor thread > - > > Key: HDFS-16484 > URL: https://issues.apache.org/jira/browse/HDFS-16484 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: qinyuren >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Attachment: 20220209-120613.html > Namenode block report add yield to avoid holding write lock too long > > > Key: HDFS-16432 > URL: https://issues.apache.org/jira/browse/HDFS-16432 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: 20220209-120613.html, image-2022-01-20-15-19-28-384.png > > Time Spent: 2h 10m > Remaining Estimate: 0h > > !image-2022-01-20-15-19-28-384.png|width=683,height=132! > In our cluster, namenode block report will held write lock for a long time if > the storage block number more than 10. So we want to add a yield > mechanism in block reporting process to avoid holding write lock too long. > # Ensure that the processing of the same block is in the same write lock. > # Because StorageInfo.addBlock will moves the block to the head of > blockList, so we can collect blocks that have not been reported by delimiter > block. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Attachment: (was: 20220209-120613.html) > Namenode block report add yield to avoid holding write lock too long > > > Key: HDFS-16432 > URL: https://issues.apache.org/jira/browse/HDFS-16432 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: image-2022-01-20-15-19-28-384.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > !image-2022-01-20-15-19-28-384.png|width=683,height=132! > In our cluster, namenode block report will held write lock for a long time if > the storage block number more than 10. So we want to add a yield > mechanism in block reporting process to avoid holding write lock too long. > # Ensure that the processing of the same block is in the same write lock. > # Because StorageInfo.addBlock will moves the block to the head of > blockList, so we can collect blocks that have not been reported by delimiter > block. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Attachment: 20220209-120613.html > Namenode block report add yield to avoid holding write lock too long > > > Key: HDFS-16432 > URL: https://issues.apache.org/jira/browse/HDFS-16432 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: 20220209-120613.html, image-2022-01-20-15-19-28-384.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > !image-2022-01-20-15-19-28-384.png|width=683,height=132! > In our cluster, namenode block report will held write lock for a long time if > the storage block number more than 10. So we want to add a yield > mechanism in block reporting process to avoid holding write lock too long. > # Ensure that the processing of the same block is in the same write lock. > # Because StorageInfo.addBlock will moves the block to the head of > blockList, so we can collect blocks that have not been reported by delimiter > block. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16423) balancer should not get blocks on stale storages
[ https://issues.apache.org/jira/browse/HDFS-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482862#comment-17482862 ] qinyuren commented on HDFS-16423: - Thanks [~sodonnell] for the advices. I will add a PR to indicate why the balancer not work after namenode failover. > balancer should not get blocks on stale storages > > > Key: HDFS-16423 > URL: https://issues.apache.org/jira/browse/HDFS-16423 > Project: Hadoop HDFS > Issue Type: Improvement > Components: balancer & mover >Reporter: qinyuren >Assignee: qinyuren >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.3 > > Attachments: image-2022-01-13-17-18-32-409.png > > Time Spent: 4h 20m > Remaining Estimate: 0h > > We have met a problems as described in HDFS-16420 > We found that balancer copied a block multi times without deleting the source > block if this block was placed in a stale storage. And resulting a block with > many copies, but these redundant copies are not deleted until the storage > become not stale. > > !image-2022-01-13-17-18-32-409.png|width=657,height=275! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17479811#comment-17479811 ] qinyuren edited comment on HDFS-16432 at 1/25/22, 6:29 AM: --- [~hexiaoqiao], [~tasanuma] Please take a look. was (Author: qinyuren): [~hexiaoqiao] [~tasanuma] Please take a look. > Namenode block report add yield to avoid holding write lock too long > > > Key: HDFS-16432 > URL: https://issues.apache.org/jira/browse/HDFS-16432 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: image-2022-01-20-15-19-28-384.png > > Time Spent: 40m > Remaining Estimate: 0h > > !image-2022-01-20-15-19-28-384.png|width=683,height=132! > In our cluster, namenode block report will held write lock for a long time if > the storage block number more than 10. So we want to add a yield > mechanism in block reporting process to avoid holding write lock too long. > # Ensure that the processing of the same block is in the same write lock. > # Because StorageInfo.addBlock will moves the block to the head of > blockList, so we can collect blocks that have not been reported by delimiter > block. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16433) The synchronized lock of IncrementalBlockReportManager will slow down the speed of creating/deleting/finalizing/opening block
[
https://issues.apache.org/jira/browse/HDFS-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480786#comment-17480786
]
qinyuren commented on HDFS-16433:
-
I think that object.wait() will release the lock.
> The synchronized lock of IncrementalBlockReportManager will slow down the
> speed of creating/deleting/finalizing/opening block
> -
>
> Key: HDFS-16433
> URL: https://issues.apache.org/jira/browse/HDFS-16433
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Yuanbo Liu
>Priority: Critical
> Attachments: image-2022-01-21-22-52-52-912.png
>
>
> The code in IncrementalBlockReportManager.java
> {code:java}
> synchronized void waitTillNextIBR(long waitTime) {
> if (waitTime > 0 && !sendImmediately()) {
> try {
> wait(ibrInterval > 0 && ibrInterval < waitTime? ibrInterval: waitTime);
> } catch (InterruptedException ie) {
> LOG.warn(getClass().getSimpleName() + " interrupted");
> }
> }
> } {code}
> We can see that wait(waitime) will hold synchronized, if ibr interval is
> enabled or heartbeat time is not reached. The lock will block
> notifyNamenodeBlock function which is widely used in
> deleting/creating/finalizing/opening blocks, then the speed of DataNode IO
> will slow down. Here is the graph of blocking relationship
> !image-2022-01-21-22-52-52-912.png|width=976,height=299!
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17479811#comment-17479811 ] qinyuren commented on HDFS-16432: - [~hexiaoqiao] [~tasanuma] Please take a look. > Namenode block report add yield to avoid holding write lock too long > > > Key: HDFS-16432 > URL: https://issues.apache.org/jira/browse/HDFS-16432 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: image-2022-01-20-15-19-28-384.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > !image-2022-01-20-15-19-28-384.png|width=683,height=132! > In our cluster, namenode block report will held write lock for a long time if > the storage block number more than 10. So we want to add a yield > mechanism in block reporting process to avoid holding write lock too long. > # Ensure that the processing of the same block is in the same write lock. > # Because StorageInfo.addBlock will moves the block to the head of > blockList, so we can collect blocks that have not been reported by delimiter > block. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Description: !image-2022-01-20-15-19-28-384.png|width=683,height=132! In our cluster, namenode block report will held write lock for a long time if the storage block number more than 10. So we want to add a yield mechanism in block reporting process to avoid holding write lock too long. # Ensure that the processing of the same block is in the same write lock. # Because StorageInfo.addBlock will moves the block to the head of blockList, so we can collect blocks that have not been reported by delimiter block. was: !image-2022-01-20-15-19-28-384.png|width=683,height=132! In our cluster, namenode block report will held write lock for a long time if the storage block number more than 10. So we want to add a yield mechanism in block reporting process to avoid holding write lock too long. > Namenode block report add yield to avoid holding write lock too long > > > Key: HDFS-16432 > URL: https://issues.apache.org/jira/browse/HDFS-16432 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: image-2022-01-20-15-19-28-384.png > > Time Spent: 10m > Remaining Estimate: 0h > > !image-2022-01-20-15-19-28-384.png|width=683,height=132! > In our cluster, namenode block report will held write lock for a long time if > the storage block number more than 10. So we want to add a yield > mechanism in block reporting process to avoid holding write lock too long. > # Ensure that the processing of the same block is in the same write lock. > # Because StorageInfo.addBlock will moves the block to the head of > blockList, so we can collect blocks that have not been reported by delimiter > block. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Description: !image-2022-01-20-15-19-28-384.png|width=683,height=132! In our cluster, namenode block report will held write lock for a long time if the storage block number more than 10. So we want to add a yield mechanism in block reporting process to avoid holding write lock too long. was: !image-2022-01-20-15-19-28-384.png|width=652,height=126! In our cluster, namenode block report will held write lock for a long time if the storage block number more than 10. So we want to add a yield mechanism in block reporting process to avoid holding write lock too long. > Namenode block report add yield to avoid holding write lock too long > > > Key: HDFS-16432 > URL: https://issues.apache.org/jira/browse/HDFS-16432 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2022-01-20-15-19-28-384.png > > > !image-2022-01-20-15-19-28-384.png|width=683,height=132! > In our cluster, namenode block report will held write lock for a long time if > the storage block number more than 10. So we want to add a yield > mechanism in block reporting process to avoid holding write lock too long. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long
[ https://issues.apache.org/jira/browse/HDFS-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16432: Description: !image-2022-01-20-15-19-28-384.png|width=652,height=126! In our cluster, namenode block report will held write lock for a long time if the storage block number more than 10. So we want to add a yield mechanism in block reporting process to avoid holding write lock too long. was:!image-2022-01-20-15-19-28-384.png|width=652,height=126! > Namenode block report add yield to avoid holding write lock too long > > > Key: HDFS-16432 > URL: https://issues.apache.org/jira/browse/HDFS-16432 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2022-01-20-15-19-28-384.png > > > !image-2022-01-20-15-19-28-384.png|width=652,height=126! > In our cluster, namenode block report will held write lock for a long time if > the storage block number more than 10. So we want to add a yield > mechanism in block reporting process to avoid holding write lock too long. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16432) Namenode block report add yield to avoid holding write lock too long
qinyuren created HDFS-16432: --- Summary: Namenode block report add yield to avoid holding write lock too long Key: HDFS-16432 URL: https://issues.apache.org/jira/browse/HDFS-16432 Project: Hadoop HDFS Issue Type: Improvement Reporter: qinyuren Attachments: image-2022-01-20-15-19-28-384.png !image-2022-01-20-15-19-28-384.png|width=652,height=126! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16423) balancer should not get blocks on stale storages
[
https://issues.apache.org/jira/browse/HDFS-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17478312#comment-17478312
]
qinyuren commented on HDFS-16423:
-
Hi [~weichiu] , the problem we described in HDFS-16420 was not related with
HDFS-12914. The problem happened because:
# Balancer copied the replica without deleting the replica from source node
after namenode HA.
# We have no set {color:#de350b}BlockPlacementPolicyRackFaultTolerant{color}
in balancer configuration, resulting in multiple copies of ec internal block
{color:#de350b}in the same rack{color}.
# Trigger namenode bug as described in
([https://github.com/apache/hadoop/pull/3880)] resulting in missing blocks.
> balancer should not get blocks on stale storages
>
>
> Key: HDFS-16423
> URL: https://issues.apache.org/jira/browse/HDFS-16423
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
>Reporter: qinyuren
>Assignee: qinyuren
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-01-13-17-18-32-409.png
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> We have met a problems as described in HDFS-16420
> We found that balancer copied a block multi times without deleting the source
> block if this block was placed in a stale storage. And resulting a block with
> many copies, but these redundant copies are not deleted until the storage
> become not stale.
>
> !image-2022-01-13-17-18-32-409.png|width=657,height=275!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17472419#comment-17472419
]
qinyuren commented on HDFS-16420:
-
Hi [~tasanuma] , we've merged these two PR into our code. It seems that they
are not related to the problem.
> ec + balancer may cause missing block
> -
>
> Key: HDFS-16420
> URL: https://issues.apache.org/jira/browse/HDFS-16420
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Attachments: image-2022-01-10-17-31-35-910.png,
> image-2022-01-10-17-32-56-981.png
>
>
> We have a similar problem as HDFS-16297 described.
> In our cluster, we used {color:#de350b}ec(6+3) + balancer with version
> 3.1.0{color}, and the {color:#de350b}missing block{color} happened.
> We got the block(blk_-9223372036824119008) info from fsck, only 5 live
> replications and multiple redundant replications.
> {code:java}
> blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
> blk_-9223372036824119007:DatanodeInfoWithStorage,
> blk_-9223372036824119002:DatanodeInfoWithStorage,
> blk_-9223372036824119001:DatanodeInfoWithStorage,
> blk_-9223372036824119000:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage {code}
>
> We searched the log from all datanode, and found that the internal blocks of
> blk_-9223372036824119008 were deleted almost at the same time.
>
> {code:java}
> 08:15:58,550 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
> file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
> 08:16:21,214 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
> file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
> 08:16:55,737 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
> file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
> {code}
>
> The total number of internal blocks deleted during 08:15-08:17 are as follows
> ||internal block||index|| delete num||
> |blk_-9223372036824119008
> blk_-9223372036824119006
> blk_-9223372036824119005
> blk_-9223372036824119004
> blk_-9223372036824119003
> blk_-9223372036824119000 |0
> 2
> 3
> 4
> 5
> 8| 1
> 1
> 1
> 50
> 1
> 1|
>
> {color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
> full block report immediately.{color}
>
> There are 2 questions:
> 1. Why are there so many replicas of this block?
> 2. Why delete the internal block with only one copy?
> The reasons for the first problem may be as follows:
> 1. We set the full block report period of some datanode to 168 hours.
> 2. We have done a namenode HA operation.
> 3. After namenode HA, the state of storage became
> {color:#ff}stale{color}, and the state not change until next full block
> report.
> 4. The balancer copied the replica without deleting the replica from source
> node, because the source node have the stale storage, and the request was put
> into {color:#ff}postponedMisreplicatedBlocks{color}.
> 5. Balancer continues to copy the replica, eventually resulting in multiple
> copies of a replica
> !image-2022-01-10-17-31-35-910.png|width=642,height=269!
> The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
> block to remove.
> !image-2022-01-10-17-32-56-981.png|width=745,height=124!
> As for the second question, we checked the code of
> {color:#de350b}processExtraRedundancyBlock{color}, but didn't find any
> problem.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17472401#comment-17472401
]
qinyuren commented on HDFS-16420:
-
Hi [~ayushtkn] , we cherry-pick most of EC fixes which merged before June 15th,
2021.
We will add detail namenode logs to track this problem.
> ec + balancer may cause missing block
> -
>
> Key: HDFS-16420
> URL: https://issues.apache.org/jira/browse/HDFS-16420
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Attachments: image-2022-01-10-17-31-35-910.png,
> image-2022-01-10-17-32-56-981.png
>
>
> We have a similar problem as HDFS-16297 described.
> In our cluster, we used {color:#de350b}ec(6+3) + balancer with version
> 3.1.0{color}, and the {color:#de350b}missing block{color} happened.
> We got the block(blk_-9223372036824119008) info from fsck, only 5 live
> replications and multiple redundant replications.
> {code:java}
> blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
> blk_-9223372036824119007:DatanodeInfoWithStorage,
> blk_-9223372036824119002:DatanodeInfoWithStorage,
> blk_-9223372036824119001:DatanodeInfoWithStorage,
> blk_-9223372036824119000:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage {code}
>
> We searched the log from all datanode, and found that the internal blocks of
> blk_-9223372036824119008 were deleted almost at the same time.
>
> {code:java}
> 08:15:58,550 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
> file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
> 08:16:21,214 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
> file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
> 08:16:55,737 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
> file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
> {code}
>
> The total number of internal blocks deleted during 08:15-08:17 are as follows
> ||internal block||index|| delete num||
> |blk_-9223372036824119008
> blk_-9223372036824119006
> blk_-9223372036824119005
> blk_-9223372036824119004
> blk_-9223372036824119003
> blk_-9223372036824119000 |0
> 2
> 3
> 4
> 5
> 8| 1
> 1
> 1
> 50
> 1
> 1|
>
> {color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
> full block report immediately.{color}
>
> There are 2 questions:
> 1. Why are there so many replicas of this block?
> 2. Why delete the internal block with only one copy?
> The reasons for the first problem may be as follows:
> 1. We set the full block report period of some datanode to 168 hours.
> 2. We have done a namenode HA operation.
> 3. After namenode HA, the state of storage became
> {color:#ff}stale{color}, and the state not change until next full block
> report.
> 4. The balancer copied the replica without deleting the replica from source
> node, because the source node have the stale storage, and the request was put
> into {color:#ff}postponedMisreplicatedBlocks{color}.
> 5. Balancer continues to copy the replica, eventually resulting in multiple
> copies of a replica
> !image-2022-01-10-17-31-35-910.png|width=642,height=269!
> The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
> block to remove.
> !image-2022-01-10-17-32-56-981.png|width=745,height=124!
> As for the second question, we checked the code of
> {color:#de350b}processExtraRedundancyBlock{color}, but didn't find any
> problem.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer with version
3.1.0{color}, and the {color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block||index|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 |0
2
3
4
5
8| 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png|width=642,height=269!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png|width=745,height=124!
As for the second question, we checked the code of
{color:#de350b}processExtraRedundancyBlock{color}, but didn't find any problem.
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer with version
3.1.0{color}, and the {color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java
[jira] [Comment Edited] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471966#comment-17471966
]
qinyuren edited comment on HDFS-16420 at 1/10/22, 12:49 PM:
Hi [~sodonnell], we cherry-pick most of EC fixes, especially those PR with data
loss. But that still doesn't solve the problem.
was (Author: qinyuren):
[~sodonnell] We cherry-pick most of EC fixes, especially those PR with data
loss.
> ec + balancer may cause missing block
> -
>
> Key: HDFS-16420
> URL: https://issues.apache.org/jira/browse/HDFS-16420
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Attachments: image-2022-01-10-17-31-35-910.png,
> image-2022-01-10-17-32-56-981.png
>
>
> We have a similar problem as HDFS-16297 described.
> In our cluster, we used {color:#de350b}ec(6+3) + balancer with version
> 3.1.0{color}, and the {color:#de350b}missing block{color} happened.
> We got the block(blk_-9223372036824119008) info from fsck, only 5 live
> replications and multiple redundant replications.
> {code:java}
> blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
> blk_-9223372036824119007:DatanodeInfoWithStorage,
> blk_-9223372036824119002:DatanodeInfoWithStorage,
> blk_-9223372036824119001:DatanodeInfoWithStorage,
> blk_-9223372036824119000:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage {code}
>
> We searched the log from all datanode, and found that the internal blocks of
> blk_-9223372036824119008 were deleted almost at the same time.
>
> {code:java}
> 08:15:58,550 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
> file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
> 08:16:21,214 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
> file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
> 08:16:55,737 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
> file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
> {code}
>
> The total number of internal blocks deleted during 08:15-08:17 are as follows
> ||internal block|| delete num||
> |blk_-9223372036824119008
> blk_-9223372036824119006
> blk_-9223372036824119005
> blk_-9223372036824119004
> blk_-9223372036824119003
> blk_-9223372036824119000 | 1
> 1
> 1
> 50
> 1
> 1|
>
> {color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
> full block report immediately.{color}
>
> There are 2 questions:
> 1. Why are there so many replicas of this block?
> 2. Why delete the internal block with only one copy?
> The reasons for the first problem may be as follows:
> 1. We set the full block report period of some datanode to 168 hours.
> 2. We have done a namenode HA operation.
> 3. After namenode HA, the state of storage became
> {color:#ff}stale{color}, and the state not change until next full block
> report.
> 4. The balancer copied the replica without deleting the replica from source
> node, because the source node have the stale storage, and the request was put
> into {color:#ff}postponedMisreplicatedBlocks{color}.
> 5. Balancer continues to copy the replica, eventually resulting in multiple
> copies of a replica
> !image-2022-01-10-17-31-35-910.png|width=642,height=269!
> The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
> block to remove.
> !image-2022-01-10-17-32-56-981.png|width=745,height=124!
> As for the second question, we checked the code of
> {color:#de350b}processExtraRedundancyBlock{color}, but didn't find any
> problem.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471966#comment-17471966
]
qinyuren commented on HDFS-16420:
-
[~sodonnell] We cherry-pick most of EC fixes, especially those PR with data
loss.
> ec + balancer may cause missing block
> -
>
> Key: HDFS-16420
> URL: https://issues.apache.org/jira/browse/HDFS-16420
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Attachments: image-2022-01-10-17-31-35-910.png,
> image-2022-01-10-17-32-56-981.png
>
>
> We have a similar problem as HDFS-16297 described.
> In our cluster, we used {color:#de350b}ec(6+3) + balancer with version
> 3.1.0{color}, and the {color:#de350b}missing block{color} happened.
> We got the block(blk_-9223372036824119008) info from fsck, only 5 live
> replications and multiple redundant replications.
> {code:java}
> blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
> blk_-9223372036824119007:DatanodeInfoWithStorage,
> blk_-9223372036824119002:DatanodeInfoWithStorage,
> blk_-9223372036824119001:DatanodeInfoWithStorage,
> blk_-9223372036824119000:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage {code}
>
> We searched the log from all datanode, and found that the internal blocks of
> blk_-9223372036824119008 were deleted almost at the same time.
>
> {code:java}
> 08:15:58,550 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
> file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
> 08:16:21,214 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
> file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
> 08:16:55,737 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
> file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
> {code}
>
> The total number of internal blocks deleted during 08:15-08:17 are as follows
> ||internal block|| delete num||
> |blk_-9223372036824119008
> blk_-9223372036824119006
> blk_-9223372036824119005
> blk_-9223372036824119004
> blk_-9223372036824119003
> blk_-9223372036824119000 | 1
> 1
> 1
> 50
> 1
> 1|
>
> {color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
> full block report immediately.{color}
>
> There are 2 questions:
> 1. Why are there so many replicas of this block?
> 2. Why delete the internal block with only one copy?
> The reasons for the first problem may be as follows:
> 1. We set the full block report period of some datanode to 168 hours.
> 2. We have done a namenode HA operation.
> 3. After namenode HA, the state of storage became
> {color:#ff}stale{color}, and the state not change until next full block
> report.
> 4. The balancer copied the replica without deleting the replica from source
> node, because the source node have the stale storage, and the request was put
> into {color:#ff}postponedMisreplicatedBlocks{color}.
> 5. Balancer continues to copy the replica, eventually resulting in multiple
> copies of a replica
> !image-2022-01-10-17-31-35-910.png|width=642,height=269!
> The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
> block to remove.
> !image-2022-01-10-17-32-56-981.png|width=745,height=124!
> As for the second question, we checked the code of
> {color:#de350b}processExtraRedundancyBlock{color}, but didn't find any
> problem.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer with version
3.1.0{color}, and the {color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png|width=642,height=269!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png|width=745,height=124!
As for the second question, we checked the code of
{color:#de350b}processExtraRedundancyBlock{color}, but didn't find any problem.
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499-xx
[jira] [Comment Edited] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471822#comment-17471822
]
qinyuren edited comment on HDFS-16420 at 1/10/22, 10:01 AM:
[~hexiaoqiao] , [~ayushtkn] Please take a look at this problem.
was (Author: qinyuren):
[~hexiaoqiao] [~ayushtkn] Please take a look at this problem.
> ec + balancer may cause missing block
> -
>
> Key: HDFS-16420
> URL: https://issues.apache.org/jira/browse/HDFS-16420
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Attachments: image-2022-01-10-17-31-35-910.png,
> image-2022-01-10-17-32-56-981.png
>
>
> We have a similar problem as HDFS-16297 described.
> In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
> {color:#de350b}missing block{color} happened.
> We got the block(blk_-9223372036824119008) info from fsck, only 5 live
> replications and multiple redundant replications.
> {code:java}
> blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
> blk_-9223372036824119007:DatanodeInfoWithStorage,
> blk_-9223372036824119002:DatanodeInfoWithStorage,
> blk_-9223372036824119001:DatanodeInfoWithStorage,
> blk_-9223372036824119000:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage {code}
>
> We searched the log from all datanode, and found that the internal blocks of
> blk_-9223372036824119008 were deleted almost at the same time.
>
> {code:java}
> 08:15:58,550 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
> file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
> 08:16:21,214 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
> file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
> 08:16:55,737 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
> file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
> {code}
>
> The total number of internal blocks deleted during 08:15-08:17 are as follows
> ||internal block|| delete num||
> |blk_-9223372036824119008
> blk_-9223372036824119006
> blk_-9223372036824119005
> blk_-9223372036824119004
> blk_-9223372036824119003
> blk_-9223372036824119000 | 1
> 1
> 1
> 50
> 1
> 1|
>
> {color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
> full block report immediately.{color}
>
> There are 2 questions:
> 1. Why are there so many replicas of this block?
> 2. Why delete the internal block with only one copy?
> The reasons for the first problem may be as follows:
> 1. We set the full block report period of some datanode to 168 hours.
> 2. We have done a namenode HA operation.
> 3. After namenode HA, the state of storage became
> {color:#ff}stale{color}, and the state not change until next full block
> report.
> 4. The balancer copied the replica without deleting the replica from source
> node, because the source node have the stale storage, and the request was put
> into {color:#ff}postponedMisreplicatedBlocks{color}.
> 5. Balancer continues to copy the replica, eventually resulting in multiple
> copies of a replica
> !image-2022-01-10-17-31-35-910.png|width=642,height=269!
> The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
> block to remove.
> !image-2022-01-10-17-32-56-981.png|width=745,height=124!
> As for the second question, we checked the code of
> {color:#de350b}processExtraRedundancyBlock{color}, but didn't find any
> problem.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17471822#comment-17471822
]
qinyuren commented on HDFS-16420:
-
[~hexiaoqiao] [~ayushtkn] Please take a look at this problem.
> ec + balancer may cause missing block
> -
>
> Key: HDFS-16420
> URL: https://issues.apache.org/jira/browse/HDFS-16420
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Attachments: image-2022-01-10-17-31-35-910.png,
> image-2022-01-10-17-32-56-981.png
>
>
> We have a similar problem as HDFS-16297 described.
> In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
> {color:#de350b}missing block{color} happened.
> We got the block(blk_-9223372036824119008) info from fsck, only 5 live
> replications and multiple redundant replications.
> {code:java}
> blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
> blk_-9223372036824119007:DatanodeInfoWithStorage,
> blk_-9223372036824119002:DatanodeInfoWithStorage,
> blk_-9223372036824119001:DatanodeInfoWithStorage,
> blk_-9223372036824119000:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage,
> blk_-9223372036824119004:DatanodeInfoWithStorage {code}
>
> We searched the log from all datanode, and found that the internal blocks of
> blk_-9223372036824119008 were deleted almost at the same time.
>
> {code:java}
> 08:15:58,550 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
> file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
> 08:16:21,214 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
> file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
> 08:16:55,737 INFO impl.FsDatasetAsyncDiskService
> (FsDatasetAsyncDiskService.java:run(333)) - Deleted
> BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
> file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
> {code}
>
> The total number of internal blocks deleted during 08:15-08:17 are as follows
> ||internal block|| delete num||
> |blk_-9223372036824119008
> blk_-9223372036824119006
> blk_-9223372036824119005
> blk_-9223372036824119004
> blk_-9223372036824119003
> blk_-9223372036824119000 | 1
> 1
> 1
> 50
> 1
> 1|
>
> {color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
> full block report immediately.{color}
>
> There are 2 questions:
> 1. Why are there so many replicas of this block?
> 2. Why delete the internal block with only one copy?
> The reasons for the first problem may be as follows:
> 1. We set the full block report period of some datanode to 168 hours.
> 2. We have done a namenode HA operation.
> 3. After namenode HA, the state of storage became
> {color:#ff}stale{color}, and the state not change until next full block
> report.
> 4. The balancer copied the replica without deleting the replica from source
> node, because the source node have the stale storage, and the request was put
> into {color:#ff}postponedMisreplicatedBlocks{color}.
> 5. Balancer continues to copy the replica, eventually resulting in multiple
> copies of a replica
> !image-2022-01-10-17-31-35-910.png|width=642,height=269!
> The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
> block to remove.
> !image-2022-01-10-17-32-56-981.png|width=745,height=124!
> As for the second question, we checked the code of
> {color:#de350b}processExtraRedundancyBlock{color}, but didn't find any
> problem.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png|width=642,height=269!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png|width=745,height=124!
As for the second question, we checked the code of
{color:#de350b}processExtraRedundancyBlock{color}, but didn't find any problem.
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 bl
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png|width=642,height=269!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png|width=745,height=124!
As for the second question, we checked the code of
{color:#de350b}processExtraRedundancyBlock{color}, but didn't find any problem.
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png|width=642,height=269!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png|width=745,height=124!
As for the second question, we checked the code of
{color:#172b4d}{color:#de350b}processExtraRedundancyBlock{color}, but didn't
find any problem.{color}
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499-x
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png|width=642,height=269!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png|width=745,height=124!
As for the second question, we checked the code of
{color:#172b4d}processExtraRedundancyBlock, but didn't find any problem.{color}
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png|width=642,height=269!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png|width=745,height=124!
As for the second question, we checked the code of
{color:#ff}processExtraRedundancyBlock{color:#172b4d}, but didn't find any
problem.{color}{color}
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499-x
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png|width=642,height=269!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png|width=745,height=124!
As for the second question, we checked the code of
{color:#ff}processExtraRedundancyBlock, but didn't find any problem.{color}
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png|width=642,height=269!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png!
As for the second question, we checked the code of
{color:#ff}processExtraRedundancyBlock, but didn't find any problem.{color}
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_2
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used {color:#de350b}ec(6+3) + balancer{color}, and the
{color:#de350b}missing block{color} happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png!
As for the second question, we checked the code of
{color:#ff}processExtraRedundancyBlock{color:#172b4d}, but didn't find any
problem.{color}{color}
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png!
As for the second question, we check the code of
{color:#ff}processExtraRedundancyBlock{color:#172b4d}, but didn't find any
problem.{color}{color}
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/curre
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#ff}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#ff}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#ff}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png!
The set of {color:#ff}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png!
As for your second question, we check the code of
{color:#FF}processExtraRedundancyBlock{color:#172b4d}, but found no
abnormalities{color}{color}
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/curren
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 | 1
1
1
50
1
1|
{color:#FF}During 08:15 to 08:17, we restarted 2 datanode and triggered
full block report immediately.{color}
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanode to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became {color:#FF}stale{color},
and the state not change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into {color:#FF}postponedMisreplicatedBlocks{color}.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png!
The set of {color:#FF}rescannedMisreplicatedBlocks{color} have so many
block to remove.
!image-2022-01-10-17-32-56-981.png!
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) -
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
||internal block|| delete num||
|blk_-9223372036824119008
blk_-9223372036824119006
blk_-9223372036824119005
blk_-9223372036824119004
blk_-9223372036824119003
blk_-9223372036824119000 |1
1
1
50
1
1|
During 08:15 to 08:17, we restarted 2 Datanodes and triggered full block
reporting immediately.
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanodes to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became stale, and the state not
change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into postponedMisreplicatedBlocks.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png!
The set of rescannedMisreplicatedBlocks have so many block to remove.
!image-2022-01-10-17-32-56-981.png!
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/d
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
{code:java}
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage {code}
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
internal block delete num
blk_-9223372036824119008 1
blk_-9223372036824119006 1
blk_-9223372036824119005 1
blk_-9223372036824119004 50
blk_-9223372036824119003 1
blk_-9223372036824119000 1
During 08:15 to 08:17, we restarted 2 Datanodes and triggered full block
reporting immediately.
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanodes to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became stale, and the state not
change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into postponedMisreplicatedBlocks.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png!
The set of rescannedMisreplicatedBlocks have so many block to remove.
!image-2022-01-10-17-32-56-981.png!
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499-
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
internal block delete num
blk_-9223372036824119008 1
blk_-9223372036824119006 1
blk_-9223372036824119005 1
blk_-9223372036824119004 50
blk_-9223372036824119003 1
blk_-9223372036824119000 1
During 08:15 to 08:17, we restarted 2 Datanodes and triggered full block
reporting immediately.
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanodes to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became stale, and the state not
change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into postponedMisreplicatedBlocks.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png!
The set of rescannedMisreplicatedBlocks have so many block to remove.
!image-2022-01-10-17-32-56-981.png!
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,521 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling
blk_-9223372036824119008_220037616 replica FinalizedReplica,
blk_-9223372036824119008_220037616, FINALIZED
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,521 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling
blk_-9223372036824119008_220037616 replica FinalizedReplica,
blk_-9223372036824119008_220037616, FINALIZED
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
internal block delete num
blk_-9223372036824119008 1
blk_-9223372036824119006 1
blk_-9223372036824119005 1
blk_-9223372036824119004 50
blk_-9223372036824119003 1
blk_-9223372036824119000 1
During 08:15 to 08:17, we restarted 2 Datanodes and triggered full block
reporting immediately.
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanodes to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became stale, and the state not
change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into postponedMisreplicatedBlocks.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png!
The set of rescannedMisreplicatedBlocks have so many block to remove.
!image-2022-01-10-17-32-56-981.png!
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,521 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling
blk_-9223372036824119008_220037616 replica FinalizedReplica,
blk_-9223372036824119008_220037616, FINALIZED
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-16
[jira] [Updated] (HDFS-16420) ec + balancer may cause missing block
[
https://issues.apache.org/jira/browse/HDFS-16420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16420:
Description:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
{code:java}
08:15:58,521 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling
blk_-9223372036824119008_220037616 replica FinalizedReplica,
blk_-9223372036824119008_220037616, FINALIZED
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI
file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008
08:16:21,214 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI
file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006
08:16:55,737 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI
file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005
{code}
The total number of internal blocks deleted during 08:15-08:17 are as follows
internal block delete num
blk_-9223372036824119008 1
blk_-9223372036824119006 1
blk_-9223372036824119005 1
blk_-9223372036824119004 50
blk_-9223372036824119003 1
blk_-9223372036824119000 1
During 08:15 to 08:17, we restarted 2 Datanodes and triggered full block
reporting immediately.
There are 2 questions:
1. Why are there so many replicas of this block?
2. Why delete the internal block with only one copy?
The reasons for the first problem may be as follows:
1. We set the full block report period of some datanodes to 168 hours.
2. We have done a namenode HA operation.
3. After namenode HA, the state of storage became stale, and the state not
change until next full block report.
4. The balancer copied the replica without deleting the replica from source
node, because the source node have the stale storage, and the request was put
into postponedMisreplicatedBlocks.
5. Balancer continues to copy the replica, eventually resulting in multiple
copies of a replica
!image-2022-01-10-17-31-35-910.png!
The set of rescannedMisreplicatedBlocks have so many block to remove.
!image-2022-01-10-17-32-56-981.png!
was:
We have a similar problem as HDFS-16297 described.
In our cluster, we used ec(6+3) + balancer, and the missing block happened.
We got the block(blk_-9223372036824119008) info from fsck, only 5 live
replications and multiple redundant replications.
blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5
blk_-9223372036824119007:DatanodeInfoWithStorage,
blk_-9223372036824119002:DatanodeInfoWithStorage,
blk_-9223372036824119001:DatanodeInfoWithStorage,
blk_-9223372036824119000:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage,
blk_-9223372036824119004:DatanodeInfoWithStorage
We searched the log from all datanode, and found that the internal blocks of
blk_-9223372036824119008 were deleted almost at the same time.
08:15:58,521 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling
blk_-9223372036824119008_220037616 replica FinalizedReplica,
blk_-9223372036824119008_220037616, FINALIZED
08:15:58,550 INFO impl.FsDatasetAsyncDiskService
(FsDatasetAsyncDiskService.java:run(333)) - Deleted
BP-1606066499--160
[jira] [Created] (HDFS-16420) ec + balancer may cause missing block
qinyuren created HDFS-16420: --- Summary: ec + balancer may cause missing block Key: HDFS-16420 URL: https://issues.apache.org/jira/browse/HDFS-16420 Project: Hadoop HDFS Issue Type: Bug Reporter: qinyuren Attachments: image-2022-01-10-17-31-35-910.png, image-2022-01-10-17-32-56-981.png We have a similar problem as HDFS-16297 described. In our cluster, we used ec(6+3) + balancer, and the missing block happened. We got the block(blk_-9223372036824119008) info from fsck, only 5 live replications and multiple redundant replications. blk_-9223372036824119008_220037616 len=133370338 MISSING! Live_repl=5 blk_-9223372036824119007:DatanodeInfoWithStorage, blk_-9223372036824119002:DatanodeInfoWithStorage, blk_-9223372036824119001:DatanodeInfoWithStorage, blk_-9223372036824119000:DatanodeInfoWithStorage, blk_-9223372036824119004:DatanodeInfoWithStorage, blk_-9223372036824119004:DatanodeInfoWithStorage, blk_-9223372036824119004:DatanodeInfoWithStorage, blk_-9223372036824119004:DatanodeInfoWithStorage, blk_-9223372036824119004:DatanodeInfoWithStorage, blk_-9223372036824119004:DatanodeInfoWithStorage We searched the log from all datanode, and found that the internal blocks of blk_-9223372036824119008 were deleted almost at the same time. 08:15:58,521 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:deleteAsync(225)) - Scheduling blk_-9223372036824119008_220037616 replica FinalizedReplica, blk_-9223372036824119008_220037616, FINALIZED 08:15:58,550 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1606066499--1606188026755 blk_-9223372036824119008_220037616 URI file:/data15/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119008 08:16:21,214 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1606066499--1606188026755 blk_-9223372036824119006_220037616 URI file:/data4/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119006 08:16:55,737 INFO impl.FsDatasetAsyncDiskService (FsDatasetAsyncDiskService.java:run(333)) - Deleted BP-1606066499--1606188026755 blk_-9223372036824119005_220037616 URI file:/data2/hadoop/hdfs/data/current/BP-1606066499--1606188026755/current/finalized/subdir19/subdir9/blk_-9223372036824119005 The total number of internal blocks deleted during 08:15-08:17 are as follows internal block delete num blk_-9223372036824119008 1 blk_-9223372036824119006 1 blk_-9223372036824119005 1 blk_-9223372036824119004 50 blk_-9223372036824119003 1 blk_-9223372036824119000 1 During 08:15 to 08:17, we restarted 2 Datanodes and triggered full block reporting immediately. There are 2 questions: 1. Why are there so many replicas of this block? 2. Why delete the internal block with only one copy? The reasons for the first problem may be as follows: 1. We set the full block report period of some datanodes to 168 hours. 2. We have done a namenode HA operation. 3. After namenode HA, the state of storage became stale, and the state not change until next full block report. 4. The balancer copied the replica without deleting the replica from source node, because the source node have the stale storage, and the request was put into postponedMisreplicatedBlocks. 5. Balancer continues to copy the replica, eventually resulting in multiple copies of a replica !image-2022-01-10-17-31-35-910.png! The set of rescannedMisreplicatedBlocks have so many block to remove. !image-2022-01-10-17-32-56-981.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16352) return the real datanode numBlocks in #getDatanodeStorageReport
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren reassigned HDFS-16352: --- Assignee: qinyuren > return the real datanode numBlocks in #getDatanodeStorageReport > --- > > Key: HDFS-16352 > URL: https://issues.apache.org/jira/browse/HDFS-16352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Assignee: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-23-22-04-06-131.png > > Time Spent: 20m > Remaining Estimate: 0h > > #getDatanodeStorageReport will return the array of DatanodeStorageReport > which contains the DatanodeInfo in each DatanodeStorageReport, but the > numBlocks in DatanodeInfo is always zero, which is confusing > !image-2021-11-23-22-04-06-131.png|width=683,height=338! > Or we can return the real numBlocks in DatanodeInfo when we call > #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16352) return the real datanode numBlocks in #getDatanodeStorageReport
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16352: Summary: return the real datanode numBlocks in #getDatanodeStorageReport (was: return the real datanode blocknums in #getDatanodeStorageReport) > return the real datanode numBlocks in #getDatanodeStorageReport > --- > > Key: HDFS-16352 > URL: https://issues.apache.org/jira/browse/HDFS-16352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2021-11-23-22-04-06-131.png > > > #getDatanodeStorageReport will return the array of DatanodeStorageReport > which contains the DatanodeInfo in each DatanodeStorageReport, but the > numBlocks in DatanodeInfo is always zero, which is confusing > !image-2021-11-23-22-04-06-131.png|width=683,height=338! > Or we can return the real numBlocks in DatanodeInfo when we call > #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16352) return the real datanode blocknums in #getDatanodeStorageReport
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16352: Summary: return the real datanode blocknums in #getDatanodeStorageReport (was: fill the real datanode blocknums in #getDatanodeStorageReport) > return the real datanode blocknums in #getDatanodeStorageReport > --- > > Key: HDFS-16352 > URL: https://issues.apache.org/jira/browse/HDFS-16352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2021-11-23-22-04-06-131.png > > > #getDatanodeStorageReport will return the array of DatanodeStorageReport > which contains the DatanodeInfo in each DatanodeStorageReport, but the > numBlocks in DatanodeInfo is always zero, which is confusing > !image-2021-11-23-22-04-06-131.png|width=683,height=338! > Or we can return the real numBlocks in DatanodeInfo when we call > #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16352) fill the real datanode blocknums in #getDatanodeStorageReport
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16352: Description: #getDatanodeStorageReport will return the array of DatanodeStorageReport which contains the DatanodeInfo in each DatanodeStorageReport, but the numBlocks in DatanodeInfo is always zero, which is confusing !image-2021-11-23-22-04-06-131.png|width=683,height=338! Or we can return the real numBlocks in DatanodeInfo when we call #getDatanodeStorageReport was: #getDatanodeStorageReport will return the array of DatanodeStorageReport which contains the DatanodeInfo in each DatanodeStorageReport, but the numBlocks in DatanodeInfo is always zero, which is confusing !image-2021-11-23-22-04-06-131.png! Or we can return the real numBlocks in DatanodeInfo when we call #getDatanodeStorageReport > fill the real datanode blocknums in #getDatanodeStorageReport > - > > Key: HDFS-16352 > URL: https://issues.apache.org/jira/browse/HDFS-16352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2021-11-23-22-04-06-131.png > > > #getDatanodeStorageReport will return the array of DatanodeStorageReport > which contains the DatanodeInfo in each DatanodeStorageReport, but the > numBlocks in DatanodeInfo is always zero, which is confusing > !image-2021-11-23-22-04-06-131.png|width=683,height=338! > Or we can return the real numBlocks in DatanodeInfo when we call > #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16352) fill the real datanode blocknums in #getDatanodeStorageReport
qinyuren created HDFS-16352: --- Summary: fill the real datanode blocknums in #getDatanodeStorageReport Key: HDFS-16352 URL: https://issues.apache.org/jira/browse/HDFS-16352 Project: Hadoop HDFS Issue Type: Improvement Reporter: qinyuren Attachments: image-2021-11-23-22-04-06-131.png #getDatanodeStorageReport will return the array of DatanodeStorageReport which contains the DatanodeInfo in each DatanodeStorageReport, but the numBlocks in DatanodeInfo is always zero, which is confusing !image-2021-11-23-22-04-06-131.png! Or we can return the real numBlocks in DatanodeInfo when we call #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
