[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[ https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-8920: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) It was commited to HDFS-7285 branch. Thanks Rui for the contribution, Colin and Zhe for the suggestions! > Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt > performance > - > > Key: HDFS-8920 > URL: https://issues.apache.org/jira/browse/HDFS-8920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-8920-HDFS-7285.1.patch, HDFS-8920-HDFS-7285.2.patch > > > When we test reading data with datanodes killed, > {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and > effectively blocks the client JVM. This log seems too verbose: > {code} > if (chosenNode == null) { > DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() + > " after checking nodes = " + Arrays.toString(nodes) + > ", ignoredNodes = " + ignoredNodes); > return null; > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[ https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-8920: Fix Version/s: HDFS-7285 > Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt > performance > - > > Key: HDFS-8920 > URL: https://issues.apache.org/jira/browse/HDFS-8920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Fix For: HDFS-7285 > > Attachments: HDFS-8920-HDFS-7285.1.patch, HDFS-8920-HDFS-7285.2.patch > > > When we test reading data with datanodes killed, > {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and > effectively blocks the client JVM. This log seems too verbose: > {code} > if (chosenNode == null) { > DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() + > " after checking nodes = " + Arrays.toString(nodes) + > ", ignoredNodes = " + ignoredNodes); > return null; > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[ https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8920: Parent Issue: HDFS-7285 (was: HDFS-8031) > Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt > performance > - > > Key: HDFS-8920 > URL: https://issues.apache.org/jira/browse/HDFS-8920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Fix For: HDFS-7285 > > Attachments: HDFS-8920-HDFS-7285.1.patch, HDFS-8920-HDFS-7285.2.patch > > > When we test reading data with datanodes killed, > {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and > effectively blocks the client JVM. This log seems too verbose: > {code} > if (chosenNode == null) { > DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() + > " after checking nodes = " + Arrays.toString(nodes) + > ", ignoredNodes = " + ignoredNodes); > return null; > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[ https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-8920: - Attachment: HDFS-8920-HDFS-7285.2.patch Address Kai's comments offline. > Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt > performance > - > > Key: HDFS-8920 > URL: https://issues.apache.org/jira/browse/HDFS-8920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-8920-HDFS-7285.1.patch, HDFS-8920-HDFS-7285.2.patch > > > When we test reading data with datanodes killed, > {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and > effectively blocks the client JVM. This log seems too verbose: > {code} > if (chosenNode == null) { > DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() + > " after checking nodes = " + Arrays.toString(nodes) + > ", ignoredNodes = " + ignoredNodes); > return null; > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[ https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-8920: - Attachment: HDFS-8920-HDFS-7285.1.patch Sorry for taking some time to get back to this. Based on my discussion with Kai offline, we think it's better to do this in {{DFSStripedInputStream}}, therefore the non-EC path won't be affected. A simple test shows that the patch reduces the log file size from 60MB to 49KB, when 5 clients read 10GB file respectively with 2 DN killed. > Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt > performance > - > > Key: HDFS-8920 > URL: https://issues.apache.org/jira/browse/HDFS-8920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-8920-HDFS-7285.1.patch > > > When we test reading data with datanodes killed, > {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and > effectively blocks the client JVM. This log seems too verbose: > {code} > if (chosenNode == null) { > DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() + > " after checking nodes = " + Arrays.toString(nodes) + > ", ignoredNodes = " + ignoredNodes); > return null; > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[ https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HDFS-8920: - Status: Patch Available (was: Open) > Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt > performance > - > > Key: HDFS-8920 > URL: https://issues.apache.org/jira/browse/HDFS-8920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > Attachments: HDFS-8920-HDFS-7285.1.patch > > > When we test reading data with datanodes killed, > {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and > effectively blocks the client JVM. This log seems too verbose: > {code} > if (chosenNode == null) { > DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() + > " after checking nodes = " + Arrays.toString(nodes) + > ", ignoredNodes = " + ignoredNodes); > return null; > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[ https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-8920: Parent Issue: HDFS-8031 (was: HDFS-7285) > Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt > performance > - > > Key: HDFS-8920 > URL: https://issues.apache.org/jira/browse/HDFS-8920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Rui Li >Assignee: Rui Li > > When we test reading data with datanodes killed, > {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and > effectively blocks the client JVM. This log seems too verbose: > {code} > if (chosenNode == null) { > DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() + > " after checking nodes = " + Arrays.toString(nodes) + > ", ignoredNodes = " + ignoredNodes); > return null; > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8920) Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance
[ https://issues.apache.org/jira/browse/HDFS-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-8920: Assignee: Rui Li Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance - Key: HDFS-8920 URL: https://issues.apache.org/jira/browse/HDFS-8920 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Rui Li Assignee: Rui Li When we test reading data with datanodes killed, {{DFSInputStream::getBestNodeDNAddrPair}} becomes a hot spot method and effectively blocks the client JVM. This log seems too verbose: {code} if (chosenNode == null) { DFSClient.LOG.warn(No live nodes contain block + block.getBlock() + after checking nodes = + Arrays.toString(nodes) + , ignoredNodes = + ignoredNodes); return null; } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)