[jira] [Updated] (HDFS-5745) Unnecessary disk check triggered when socket operation has problem.
[
https://issues.apache.org/jira/browse/HDFS-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-5745:
---
Labels: (was: BB2015-05-RFC)
> Unnecessary disk check triggered when socket operation has problem.
> ---
>
> Key: HDFS-5745
> URL: https://issues.apache.org/jira/browse/HDFS-5745
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.2.1
>Reporter: MaoYuan Xian
>Assignee: jun aoki
> Attachments: HDFS-5745.patch
>
>
> When BlockReceiver transfer data fails, it can be found SocketOutputStream
> translates the exception as IOException with the message "The stream is
> closed":
> 2014-01-06 11:48:04,716 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in BlockReceiver.run():
> java.io.IOException: The stream is closed
> at org.apache.hadoop.net.SocketOutputStream.write
> at java.io.BufferedOutputStream.flushBuffer
> at java.io.BufferedOutputStream.flush
> at java.io.DataOutputStream.flush
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run
> at java.lang.Thread.run
> Which makes the checkDiskError method of DataNode called and triggers the
> disk scan.
> Can we make the modifications like below in checkDiskError to avoiding this
> unneccessary disk scan operations?:
> {code}
> --- a/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> +++ b/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> @@ -938,7 +938,8 @@ public class DataNode extends Configured
> || e.getMessage().startsWith("An established connection was
> aborted")
> || e.getMessage().startsWith("Broken pipe")
> || e.getMessage().startsWith("Connection reset")
> - || e.getMessage().contains("java.nio.channels.SocketChannel")) {
> + || e.getMessage().contains("java.nio.channels.SocketChannel")
> + || e.getMessage().startsWith("The stream is closed")) {
>LOG.info("Not checking disk as checkDiskError was called on a network"
> +
> " related exception");
>return;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-5745) Unnecessary disk check triggered when socket operation has problem.
[
https://issues.apache.org/jira/browse/HDFS-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jun aoki updated HDFS-5745:
---
Labels: BB2015-05-RFC (was: )
> Unnecessary disk check triggered when socket operation has problem.
> ---
>
> Key: HDFS-5745
> URL: https://issues.apache.org/jira/browse/HDFS-5745
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.2.1
>Reporter: MaoYuan Xian
>Assignee: jun aoki
> Labels: BB2015-05-RFC
> Attachments: HDFS-5745.patch
>
>
> When BlockReceiver transfer data fails, it can be found SocketOutputStream
> translates the exception as IOException with the message "The stream is
> closed":
> 2014-01-06 11:48:04,716 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in BlockReceiver.run():
> java.io.IOException: The stream is closed
> at org.apache.hadoop.net.SocketOutputStream.write
> at java.io.BufferedOutputStream.flushBuffer
> at java.io.BufferedOutputStream.flush
> at java.io.DataOutputStream.flush
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run
> at java.lang.Thread.run
> Which makes the checkDiskError method of DataNode called and triggers the
> disk scan.
> Can we make the modifications like below in checkDiskError to avoiding this
> unneccessary disk scan operations?:
> {code}
> --- a/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> +++ b/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> @@ -938,7 +938,8 @@ public class DataNode extends Configured
> || e.getMessage().startsWith("An established connection was
> aborted")
> || e.getMessage().startsWith("Broken pipe")
> || e.getMessage().startsWith("Connection reset")
> - || e.getMessage().contains("java.nio.channels.SocketChannel")) {
> + || e.getMessage().contains("java.nio.channels.SocketChannel")
> + || e.getMessage().startsWith("The stream is closed")) {
>LOG.info("Not checking disk as checkDiskError was called on a network"
> +
> " related exception");
>return;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-5745) Unnecessary disk check triggered when socket operation has problem.
[
https://issues.apache.org/jira/browse/HDFS-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jun aoki updated HDFS-5745:
---
Status: Open (was: Patch Available)
> Unnecessary disk check triggered when socket operation has problem.
> ---
>
> Key: HDFS-5745
> URL: https://issues.apache.org/jira/browse/HDFS-5745
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.2.1
>Reporter: MaoYuan Xian
>Assignee: jun aoki
> Attachments: HDFS-5745.patch
>
>
> When BlockReceiver transfer data fails, it can be found SocketOutputStream
> translates the exception as IOException with the message "The stream is
> closed":
> 2014-01-06 11:48:04,716 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in BlockReceiver.run():
> java.io.IOException: The stream is closed
> at org.apache.hadoop.net.SocketOutputStream.write
> at java.io.BufferedOutputStream.flushBuffer
> at java.io.BufferedOutputStream.flush
> at java.io.DataOutputStream.flush
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run
> at java.lang.Thread.run
> Which makes the checkDiskError method of DataNode called and triggers the
> disk scan.
> Can we make the modifications like below in checkDiskError to avoiding this
> unneccessary disk scan operations?:
> {code}
> --- a/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> +++ b/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> @@ -938,7 +938,8 @@ public class DataNode extends Configured
> || e.getMessage().startsWith("An established connection was
> aborted")
> || e.getMessage().startsWith("Broken pipe")
> || e.getMessage().startsWith("Connection reset")
> - || e.getMessage().contains("java.nio.channels.SocketChannel")) {
> + || e.getMessage().contains("java.nio.channels.SocketChannel")
> + || e.getMessage().startsWith("The stream is closed")) {
>LOG.info("Not checking disk as checkDiskError was called on a network"
> +
> " related exception");
>return;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-5745) Unnecessary disk check triggered when socket operation has problem.
[
https://issues.apache.org/jira/browse/HDFS-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jun aoki updated HDFS-5745:
---
Labels: (was: BB2015-05-TBR)
> Unnecessary disk check triggered when socket operation has problem.
> ---
>
> Key: HDFS-5745
> URL: https://issues.apache.org/jira/browse/HDFS-5745
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.2.1
>Reporter: MaoYuan Xian
>Assignee: jun aoki
> Attachments: HDFS-5745.patch
>
>
> When BlockReceiver transfer data fails, it can be found SocketOutputStream
> translates the exception as IOException with the message "The stream is
> closed":
> 2014-01-06 11:48:04,716 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in BlockReceiver.run():
> java.io.IOException: The stream is closed
> at org.apache.hadoop.net.SocketOutputStream.write
> at java.io.BufferedOutputStream.flushBuffer
> at java.io.BufferedOutputStream.flush
> at java.io.DataOutputStream.flush
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run
> at java.lang.Thread.run
> Which makes the checkDiskError method of DataNode called and triggers the
> disk scan.
> Can we make the modifications like below in checkDiskError to avoiding this
> unneccessary disk scan operations?:
> {code}
> --- a/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> +++ b/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> @@ -938,7 +938,8 @@ public class DataNode extends Configured
> || e.getMessage().startsWith("An established connection was
> aborted")
> || e.getMessage().startsWith("Broken pipe")
> || e.getMessage().startsWith("Connection reset")
> - || e.getMessage().contains("java.nio.channels.SocketChannel")) {
> + || e.getMessage().contains("java.nio.channels.SocketChannel")
> + || e.getMessage().startsWith("The stream is closed")) {
>LOG.info("Not checking disk as checkDiskError was called on a network"
> +
> " related exception");
>return;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-5745) Unnecessary disk check triggered when socket operation has problem.
[
https://issues.apache.org/jira/browse/HDFS-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated HDFS-5745:
---
Labels: BB2015-05-TBR (was: )
> Unnecessary disk check triggered when socket operation has problem.
> ---
>
> Key: HDFS-5745
> URL: https://issues.apache.org/jira/browse/HDFS-5745
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.2.1
>Reporter: MaoYuan Xian
> Labels: BB2015-05-TBR
> Attachments: HDFS-5745.patch
>
>
> When BlockReceiver transfer data fails, it can be found SocketOutputStream
> translates the exception as IOException with the message "The stream is
> closed":
> 2014-01-06 11:48:04,716 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in BlockReceiver.run():
> java.io.IOException: The stream is closed
> at org.apache.hadoop.net.SocketOutputStream.write
> at java.io.BufferedOutputStream.flushBuffer
> at java.io.BufferedOutputStream.flush
> at java.io.DataOutputStream.flush
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run
> at java.lang.Thread.run
> Which makes the checkDiskError method of DataNode called and triggers the
> disk scan.
> Can we make the modifications like below in checkDiskError to avoiding this
> unneccessary disk scan operations?:
> {code}
> --- a/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> +++ b/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> @@ -938,7 +938,8 @@ public class DataNode extends Configured
> || e.getMessage().startsWith("An established connection was
> aborted")
> || e.getMessage().startsWith("Broken pipe")
> || e.getMessage().startsWith("Connection reset")
> - || e.getMessage().contains("java.nio.channels.SocketChannel")) {
> + || e.getMessage().contains("java.nio.channels.SocketChannel")
> + || e.getMessage().startsWith("The stream is closed")) {
>LOG.info("Not checking disk as checkDiskError was called on a network"
> +
> " related exception");
>return;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-5745) Unnecessary disk check triggered when socket operation has problem.
[
https://issues.apache.org/jira/browse/HDFS-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MaoYuan Xian updated HDFS-5745:
---
Status: Patch Available (was: Open)
> Unnecessary disk check triggered when socket operation has problem.
> ---
>
> Key: HDFS-5745
> URL: https://issues.apache.org/jira/browse/HDFS-5745
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.2.1
>Reporter: MaoYuan Xian
> Attachments: HDFS-5745.patch
>
>
> When BlockReceiver transfer data fails, it can be found SocketOutputStream
> translates the exception as IOException with the message "The stream is
> closed":
> 2014-01-06 11:48:04,716 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in BlockReceiver.run():
> java.io.IOException: The stream is closed
> at org.apache.hadoop.net.SocketOutputStream.write
> at java.io.BufferedOutputStream.flushBuffer
> at java.io.BufferedOutputStream.flush
> at java.io.DataOutputStream.flush
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run
> at java.lang.Thread.run
> Which makes the checkDiskError method of DataNode called and triggers the
> disk scan.
> Can we make the modifications like below in checkDiskError to avoiding this
> unneccessary disk scan operations?:
> {code}
> --- a/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> +++ b/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> @@ -938,7 +938,8 @@ public class DataNode extends Configured
> || e.getMessage().startsWith("An established connection was
> aborted")
> || e.getMessage().startsWith("Broken pipe")
> || e.getMessage().startsWith("Connection reset")
> - || e.getMessage().contains("java.nio.channels.SocketChannel")) {
> + || e.getMessage().contains("java.nio.channels.SocketChannel")
> + || e.getMessage().startsWith("The stream is closed")) {
>LOG.info("Not checking disk as checkDiskError was called on a network"
> +
> " related exception");
>return;
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5745) Unnecessary disk check triggered when socket operation has problem.
[
https://issues.apache.org/jira/browse/HDFS-5745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MaoYuan Xian updated HDFS-5745:
---
Attachment: HDFS-5745.patch
> Unnecessary disk check triggered when socket operation has problem.
> ---
>
> Key: HDFS-5745
> URL: https://issues.apache.org/jira/browse/HDFS-5745
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 1.2.1
>Reporter: MaoYuan Xian
> Attachments: HDFS-5745.patch
>
>
> When BlockReceiver transfer data fails, it can be found SocketOutputStream
> translates the exception as IOException with the message "The stream is
> closed":
> 2014-01-06 11:48:04,716 WARN org.apache.hadoop.hdfs.server.datanode.DataNode:
> IOException in BlockReceiver.run():
> java.io.IOException: The stream is closed
> at org.apache.hadoop.net.SocketOutputStream.write
> at java.io.BufferedOutputStream.flushBuffer
> at java.io.BufferedOutputStream.flush
> at java.io.DataOutputStream.flush
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run
> at java.lang.Thread.run
> Which makes the checkDiskError method of DataNode called and triggers the
> disk scan.
> Can we make the modifications like below in checkDiskError to avoiding this
> unneccessary disk scan operations?:
> {code}
> --- a/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> +++ b/src/hdfs/org/apache/hadoop/hdfs/server/datanode/DataNode.java
> @@ -938,7 +938,8 @@ public class DataNode extends Configured
> || e.getMessage().startsWith("An established connection was
> aborted")
> || e.getMessage().startsWith("Broken pipe")
> || e.getMessage().startsWith("Connection reset")
> - || e.getMessage().contains("java.nio.channels.SocketChannel")) {
> + || e.getMessage().contains("java.nio.channels.SocketChannel")
> + || e.getMessage().startsWith("The stream is closed")) {
>LOG.info("Not checking disk as checkDiskError was called on a network"
> +
> " related exception");
>return;
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
