[jira] [Commented] (HDFS-8762) Erasure Coding: the log of each streamer should show its index

2015-08-04 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653299#comment-14653299 ] Li Bo commented on HDFS-8762: - Move this jira from HDFS-8031 to HDFS-7285. Because there're 9

[jira] [Commented] (HDFS-8762) Erasure Coding: the log of each streamer should show its index

2015-08-04 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1465#comment-1465 ] Li Bo commented on HDFS-8762: - hi, [~zhz] The Jenkins test of HDFS-8704 fails, but I can't

[jira] [Updated] (HDFS-8762) Erasure Coding: the log of each streamer should show its index

2015-08-04 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8762: Parent Issue: HDFS-7285 (was: HDFS-8031) Erasure Coding: the log of each streamer should show its index

[jira] [Updated] (HDFS-8762) Erasure Coding: the log of each streamer should show its index

2015-08-04 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8762: Attachment: HDFS-8762-HDFS-7285-002.patch Erasure Coding: the log of each streamer should show its index

[jira] [Created] (HDFS-8857) Erasure Coding: Fix ArrayIndexOutOfBoundsException in TestWriteStripedFileWithFailure

2015-08-04 Thread Li Bo (JIRA)
Li Bo created HDFS-8857: --- Summary: Erasure Coding: Fix ArrayIndexOutOfBoundsException in TestWriteStripedFileWithFailure Key: HDFS-8857 URL: https://issues.apache.org/jira/browse/HDFS-8857 Project: Hadoop HDFS

[jira] [Updated] (HDFS-8857) Erasure Coding: Fix ArrayIndexOutOfBoundsException in TestWriteStripedFileWithFailure

2015-08-04 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8857: Attachment: HDFS-8857-HDFS-7285-001.patch Erasure Coding: Fix ArrayIndexOutOfBoundsException in

[jira] [Updated] (HDFS-8857) Erasure Coding: Fix ArrayIndexOutOfBoundsException in TestWriteStripedFileWithFailure

2015-08-04 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8857: Status: Patch Available (was: Open) Erasure Coding: Fix ArrayIndexOutOfBoundsException in

[jira] [Updated] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-08-03 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8704: Attachment: HDFS-8704-HDFS-7285-004.patch Erasure Coding: client fails to write large file when one datanode fails

[jira] [Updated] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-08-03 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8704: Status: Patch Available (was: Open) Erasure Coding: client fails to write large file when one datanode fails

[jira] [Commented] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small

2015-08-03 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14651555#comment-14651555 ] Li Bo commented on HDFS-8838: - The number of datanodes is set to 9 in the unit test. Due to the

[jira] [Commented] (HDFS-8839) Erasure Coding: client occasionally gets less block locations when some datanodes fail

2015-08-02 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14651460#comment-14651460 ] Li Bo commented on HDFS-8839: - hi, Zhe, I remember HDFS-8820 handles the situation that a

[jira] [Commented] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small

2015-07-31 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648802#comment-14648802 ] Li Bo commented on HDFS-8838: - hi, Nicholas I think you can commit your patch first and I will

[jira] [Commented] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-07-30 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647239#comment-14647239 ] Li Bo commented on HDFS-8704: - I have just update a second patch of this problem. The changes

[jira] [Commented] (HDFS-8838) Tolerate datanode failures in DFSStripedOutputStream when the data length is small

2015-07-30 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647366#comment-14647366 ] Li Bo commented on HDFS-8838: - Thanks Nicholas for the effort on this problem. I have just

[jira] [Updated] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-07-30 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8704: Attachment: HDFS-8704-HDFS-7285-003.patch Erasure Coding: client fails to write large file when one datanode fails

[jira] [Commented] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-07-30 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647363#comment-14647363 ] Li Bo commented on HDFS-8704: - Patch 003 fixes the problem of writing a file that is a little

[jira] [Commented] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-07-29 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647223#comment-14647223 ] Li Bo commented on HDFS-8704: - Thanks for reporting this issue. This bug is really complex and

[jira] [Created] (HDFS-8839) Erasure Coding: client occasionally gets less block locations when some datanodes fail

2015-07-29 Thread Li Bo (JIRA)
Li Bo created HDFS-8839: --- Summary: Erasure Coding: client occasionally gets less block locations when some datanodes fail Key: HDFS-8839 URL: https://issues.apache.org/jira/browse/HDFS-8839 Project: Hadoop

[jira] [Updated] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-07-29 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8704: Attachment: HDFS-8704-HDFS-7285-002.patch Erasure Coding: client fails to write large file when one datanode fails

[jira] [Resolved] (HDFS-8313) Erasure Coding: DFSStripedOutputStream#close throws NullPointerException exception in some cases

2015-07-15 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo resolved HDFS-8313. - Resolution: Cannot Reproduce Erasure Coding: DFSStripedOutputStream#close throws NullPointerException exception

[jira] [Commented] (HDFS-8762) Erasure Coding: the log of each streamer should show its index

2015-07-15 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627666#comment-14627666 ] Li Bo commented on HDFS-8762: - Add {{this}} to log string is also a solution. But it adds too

[jira] [Commented] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-07-15 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14627677#comment-14627677 ] Li Bo commented on HDFS-8704: - I am still working on this jira. The error is random and now it

[jira] [Updated] (HDFS-8762) Erasure Coding: the log of each streamer should show its index

2015-07-15 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8762: Status: Patch Available (was: Open) Erasure Coding: the log of each streamer should show its index

[jira] [Updated] (HDFS-8762) Erasure Coding: the log of each streamer should show its index

2015-07-15 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8762: Attachment: HDFS-8762-HDFS-7285-001.patch Erasure Coding: the log of each streamer should show its index

[jira] [Commented] (HDFS-8734) Erasure Coding: fix one cell need two packets

2015-07-13 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14625852#comment-14625852 ] Li Bo commented on HDFS-8734: - Thanks for Walter's effort on this jira. Please see my review as

[jira] [Updated] (HDFS-8744) Erasure Coding: the number of chunks in packet is not updated when writing parity data

2015-07-12 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8744: Resolution: Fixed Status: Resolved (was: Patch Available) Erasure Coding: the number of chunks in packet

[jira] [Created] (HDFS-8762) Erasure Coding: the log of each streamer should show its index

2015-07-12 Thread Li Bo (JIRA)
Li Bo created HDFS-8762: --- Summary: Erasure Coding: the log of each streamer should show its index Key: HDFS-8762 URL: https://issues.apache.org/jira/browse/HDFS-8762 Project: Hadoop HDFS Issue Type:

[jira] [Updated] (HDFS-8744) Erasure Coding: the number of chunks in packet is not updated when writing parity data

2015-07-09 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8744: Attachment: HDFS-8744-HDFS-7285-002.patch update patch according to Jing's review Erasure Coding: the number of

[jira] [Updated] (HDFS-8744) Erasure Coding: the number of chunks in packet is not updated when writing parity data

2015-07-09 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8744: Description: The member {{numChunks}} in {{DFSPacket}} is always zero if this packet contains parity data. The

[jira] [Updated] (HDFS-8744) Erasure Coding: the number of chunks in packet is not updated when writing parity data

2015-07-09 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8744: Description: The member {{numChunks}} in {{DFSPacket}} is always zero if this packet contains parity data. The

[jira] [Updated] (HDFS-8744) Erasure Coding: the number of chunks in packet is not updated when writing parity data

2015-07-09 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8744: Status: Patch Available (was: Open) Erasure Coding: the number of chunks in packet is not updated when writing

[jira] [Created] (HDFS-8744) Erasure Coding: the number of chunks in packet is not updated when writing parity data

2015-07-09 Thread Li Bo (JIRA)
Li Bo created HDFS-8744: --- Summary: Erasure Coding: the number of chunks in packet is not updated when writing parity data Key: HDFS-8744 URL: https://issues.apache.org/jira/browse/HDFS-8744 Project: Hadoop

[jira] [Commented] (HDFS-8719) Erasure Coding: client generates too many small packets when writing parity data

2015-07-09 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620109#comment-14620109 ] Li Bo commented on HDFS-8719: - Thanks [~jingzhao] and [~walter.k.su]! Erasure Coding: client

[jira] [Updated] (HDFS-8744) Erasure Coding: the number of chunks in packet is not updated when writing parity data

2015-07-09 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8744: Attachment: HDFS-8744-HDFS-7285-001.patch Erasure Coding: the number of chunks in packet is not updated when

[jira] [Commented] (HDFS-8719) Erasure Coding: client generates too many small packets when writing parity data

2015-07-08 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618249#comment-14618249 ] Li Bo commented on HDFS-8719: - Yes, when to start a parity cell, the {{streamer}} used in

[jira] [Updated] (HDFS-8719) Erasure Coding: client generates too many small packets when writing parity data

2015-07-08 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8719: Attachment: HDFS-8719-HDFS-7285-002.patch Erasure Coding: client generates too many small packets when writing

[jira] [Commented] (HDFS-8719) Erasure Coding: client generates too many small packets when writing parity data

2015-07-08 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618144#comment-14618144 ] Li Bo commented on HDFS-8719: - Thanks Walter for the review. {{remainingBytes}} will be not 1

[jira] [Updated] (HDFS-8719) Erasure Coding: client generates too many small packets when writing parity data

2015-07-08 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8719: Attachment: HDFS-8719-HDFS-7285-001.patch Erasure Coding: client generates too many small packets when writing

[jira] [Commented] (HDFS-8734) Erasure Coding: fix one cell need two packets

2015-07-08 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619812#comment-14619812 ] Li Bo commented on HDFS-8734: - We know that the data of a full packet is a little smaller than

[jira] [Updated] (HDFS-8719) Erasure Coding: client generates too many small packets when writing parity data

2015-07-08 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8719: Attachment: HDFS-8719-HDFS-7285-003.patch {{adjustChunkBoundary()}} is move to {{setCurrentStreamer}}.

[jira] [Created] (HDFS-8719) Erasure Coding: client generates too many small packets when writing parity data

2015-07-07 Thread Li Bo (JIRA)
Li Bo created HDFS-8719: --- Summary: Erasure Coding: client generates too many small packets when writing parity data Key: HDFS-8719 URL: https://issues.apache.org/jira/browse/HDFS-8719 Project: Hadoop HDFS

[jira] [Updated] (HDFS-8719) Erasure Coding: client generates too many small packets when writing parity data

2015-07-07 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8719: Attachment: HDFS-8719-001.patch Erasure Coding: client generates too many small packets when writing parity data

[jira] [Updated] (HDFS-8719) Erasure Coding: client generates too many small packets when writing parity data

2015-07-07 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8719: Status: Patch Available (was: Open) Erasure Coding: client generates too many small packets when writing parity

[jira] [Created] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-07-01 Thread Li Bo (JIRA)
Li Bo created HDFS-8704: --- Summary: Erasure Coding: client fails to write large file when one datanode fails Key: HDFS-8704 URL: https://issues.apache.org/jira/browse/HDFS-8704 Project: Hadoop HDFS

[jira] [Updated] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-07-01 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8704: Attachment: HDFS-8704-000.patch Erasure Coding: client fails to write large file when one datanode fails

[jira] [Updated] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-07-01 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8704: Description: I test current code on a 5-node cluster using RS(3,2). When a datanode is corrupt, client succeeds to

[jira] [Commented] (HDFS-8704) Erasure Coding: client fails to write large file when one datanode fails

2015-07-01 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14611387#comment-14611387 ] Li Bo commented on HDFS-8704: - Oh, I see, HDFS-8254 had discussed this problem before its

[jira] [Commented] (HDFS-8313) Erasure Coding: DFSStripedOutputStream#close throws NullPointerException exception in some cases

2015-06-28 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14605032#comment-14605032 ] Li Bo commented on HDFS-8313: - Sorry for the late reply due to my annual leaves these days. The

[jira] [Updated] (HDFS-8529) Add block count metrics to datanode for EC recovery and encoding

2015-06-09 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8529: Attachment: HDFS-8529-001.patch Add block count metrics to datanode for EC recovery and encoding

[jira] [Commented] (HDFS-8287) DFSStripedOutputStream.writeChunk should not wait for writing parity

2015-06-04 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14573971#comment-14573971 ] Li Bo commented on HDFS-8287: - hi, Nicholas, have you started this sub task?

[jira] [Updated] (HDFS-8411) Add to Datanode the amount of data that is read and transfered during EC reconstruction

2015-06-03 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8411: Attachment: HDFS-8411-001.patch Add to Datanode the amount of data that is read and transfered during EC

[jira] [Updated] (HDFS-8410) Add to Datanode the time taken for computations during EC reconstruction

2015-06-03 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8410: Attachment: HDFS-8410-001.patch Add to Datanode the time taken for computations during EC reconstruction

[jira] [Commented] (HDFS-8410) Add to Datanode the time taken for computations during EC reconstruction

2015-06-03 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570331#comment-14570331 ] Li Bo commented on HDFS-8410: - The patch is based on HDFS-8449 Add to Datanode the time taken

[jira] [Updated] (HDFS-8529) Add block count metrics to datanode for EC recovery and encoding

2015-06-03 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8529: Summary: Add block count metrics to datanode for EC recovery and encoding (was: Erasure Coding: optimize client

[jira] [Updated] (HDFS-8529) Add block count metrics to datanode for EC recovery and encoding

2015-06-03 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8529: Description: This sub task will add block count metrics to datanode that takes the encoding and recovery tasks.

[jira] [Created] (HDFS-8528) Erasure Coding: optimize client writing by making the writing of data and parity concurrently

2015-06-03 Thread Li Bo (JIRA)
Li Bo created HDFS-8528: --- Summary: Erasure Coding: optimize client writing by making the writing of data and parity concurrently Key: HDFS-8528 URL: https://issues.apache.org/jira/browse/HDFS-8528 Project:

[jira] [Commented] (HDFS-8425) [umbrella] Performance tuning, investigation and optimization for erasure coding

2015-06-03 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572086#comment-14572086 ] Li Bo commented on HDFS-8425: - I think encoding is one factor. More work can be done to enhance

[jira] [Created] (HDFS-8529) Erasure Coding: optimize client writing by making the writing of data and parity concurrently

2015-06-03 Thread Li Bo (JIRA)
Li Bo created HDFS-8529: --- Summary: Erasure Coding: optimize client writing by making the writing of data and parity concurrently Key: HDFS-8529 URL: https://issues.apache.org/jira/browse/HDFS-8529 Project:

[jira] [Commented] (HDFS-8497) ErasureCodingWorker fails to do decode work

2015-06-02 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570218#comment-14570218 ] Li Bo commented on HDFS-8497: - Sorry for the late reply due to my annual leave from work. I run

[jira] [Updated] (HDFS-8497) ErasureCodingWorker fails to do decode work

2015-05-29 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8497: Attachment: HDFS-8497-HDFS-7285-01.patch The unit test is another test for recovery work of datanode, I think we can

[jira] [Commented] (HDFS-8497) ErasureCodingWorker fails to do decode work

2015-05-29 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14564252#comment-14564252 ] Li Bo commented on HDFS-8497: - error correct: before HDFS-8449 ErasureCodingWorker fails to

[jira] [Created] (HDFS-8497) ErasureCodingWorker fails to do decode work

2015-05-28 Thread Li Bo (JIRA)
Li Bo created HDFS-8497: --- Summary: ErasureCodingWorker fails to do decode work Key: HDFS-8497 URL: https://issues.apache.org/jira/browse/HDFS-8497 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Updated] (HDFS-8449) Add ec recovery tasks count metric to DataNode

2015-05-22 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8449: Attachment: HDFS-8449-001.patch Add ec recovery tasks count metric to DataNode

[jira] [Created] (HDFS-8449) Add ec recovery tasks count metric to DataNode

2015-05-20 Thread Li Bo (JIRA)
Li Bo created HDFS-8449: --- Summary: Add ec recovery tasks count metric to DataNode Key: HDFS-8449 URL: https://issues.apache.org/jira/browse/HDFS-8449 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Updated] (HDFS-8449) Add ec recovery tasks count metric to DataNode

2015-05-20 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8449: Attachment: HDFS-8449-000.patch Add ec recovery tasks count metric to DataNode

[jira] [Commented] (HDFS-8366) Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream

2015-05-19 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550006#comment-14550006 ] Li Bo commented on HDFS-8366: - Hi, Nicholas Different streamers may have different write speed,

[jira] [Updated] (HDFS-8366) Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream

2015-05-19 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8366: Resolution: Fixed Status: Resolved (was: Patch Available) Erasure Coding: Make the timeout parameter of

[jira] [Created] (HDFS-8410) Add to Datanode the time taken for computations during EC reconstruction

2015-05-15 Thread Li Bo (JIRA)
Li Bo created HDFS-8410: --- Summary: Add to Datanode the time taken for computations during EC reconstruction Key: HDFS-8410 URL: https://issues.apache.org/jira/browse/HDFS-8410 Project: Hadoop HDFS

[jira] [Assigned] (HDFS-8410) Add to Datanode the time taken for computations during EC reconstruction

2015-05-15 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo reassigned HDFS-8410: --- Assignee: Li Bo Add to Datanode the time taken for computations during EC reconstruction

[jira] [Commented] (HDFS-8366) Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream

2015-05-15 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14544984#comment-14544984 ] Li Bo commented on HDFS-8366: - Thanks for Kai's careful review. I will commit it later.

[jira] [Created] (HDFS-8411) Add to Datanode the amount of data that is read and transfered during EC reconstruction

2015-05-15 Thread Li Bo (JIRA)
Li Bo created HDFS-8411: --- Summary: Add to Datanode the amount of data that is read and transfered during EC reconstruction Key: HDFS-8411 URL: https://issues.apache.org/jira/browse/HDFS-8411 Project: Hadoop

[jira] [Updated] (HDFS-8366) Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream

2015-05-14 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8366: Attachment: HDFS-8366-HDFS-7285-02.patch Erasure Coding: Make the timeout parameter of polling blocking queue

[jira] [Commented] (HDFS-7674) Adding metrics for Erasure Coding

2015-05-13 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541536#comment-14541536 ] Li Bo commented on HDFS-7674: - hi, Wei, could you assign this sub task to me ? Adding metrics

[jira] [Assigned] (HDFS-7674) Adding metrics for Erasure Coding

2015-05-13 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo reassigned HDFS-7674: --- Assignee: Li Bo (was: Wei Zhou) Adding metrics for Erasure Coding -

[jira] [Updated] (HDFS-8366) Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream

2015-05-12 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8366: Status: Patch Available (was: Open) Erasure Coding: Make the timeout parameter of polling blocking queue

[jira] [Commented] (HDFS-8220) Erasure Coding: StripedDataStreamer fails to handle the blocklocations which doesn't satisfy BlockGroupSize

2015-05-11 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537636#comment-14537636 ] Li Bo commented on HDFS-8220: - hi, Rakesh Sorry for the late reply. I notice HDFS-8365 has

[jira] [Created] (HDFS-8366) Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream

2015-05-11 Thread Li Bo (JIRA)
Li Bo created HDFS-8366: --- Summary: Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream Key: HDFS-8366 URL: https://issues.apache.org/jira/browse/HDFS-8366

[jira] [Updated] (HDFS-8366) Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream

2015-05-11 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8366: Description: The timeout of getting striped or ended block in {{DFSStripedOutputStream#Coodinator}} should be

[jira] [Updated] (HDFS-8366) Erasure Coding: Make the timeout parameter of polling blocking queue configurable in DFSStripedOutputStream

2015-05-11 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8366: Attachment: HDFS-8366-001.patch Erasure Coding: Make the timeout parameter of polling blocking queue configurable

[jira] [Commented] (HDFS-8313) Erasure Coding: DFSStripedOutputStream#close throws NullPointerException exception in some cases

2015-05-03 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526189#comment-14526189 ] Li Bo commented on HDFS-8313: - hi, Yi, could you assign this task to me? Erasure Coding:

[jira] [Commented] (HDFS-7348) Erasure Coding: striped block recovery

2015-04-30 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520951#comment-14520951 ] Li Bo commented on HDFS-7348: - bq. We can do local writing and local reading logics as

[jira] [Updated] (HDFS-8015) Erasure Coding: local and remote block writer for coding work in DataNode

2015-04-29 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8015: Attachment: HDFS-8015-001.patch Erasure Coding: local and remote block writer for coding work in DataNode

[jira] [Commented] (HDFS-7348) Erasure Coding: striped block recovery

2015-04-29 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518935#comment-14518935 ] Li Bo commented on HDFS-7348: - We can give two working modes to the recovery work, Fast, or

[jira] [Commented] (HDFS-7613) Block placement policy for erasure coding groups

2015-04-29 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518818#comment-14518818 ] Li Bo commented on HDFS-7613: - hi, Walter HDFS-8220 depends upon this sub task but the current

[jira] [Commented] (HDFS-7348) Erasure Coding: striped block recovery

2015-04-29 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518918#comment-14518918 ] Li Bo commented on HDFS-7348: - Thanks Yi for the great work! I think decreasing network I/O is

[jira] [Commented] (HDFS-8220) Erasure Coding: StripedDataStreamer fails to handle the blocklocations which doesn't satisfy BlockGroupSize

2015-04-28 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516627#comment-14516627 ] Li Bo commented on HDFS-8220: - My suggestions are as follows: 1. We can add

[jira] [Commented] (HDFS-8220) Erasure Coding: StripedDataStreamer fails to handle the blocklocations which doesn't satisfy BlockGroupSize

2015-04-28 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516512#comment-14516512 ] Li Bo commented on HDFS-8220: - hi, Rakesh Thanks for finding and fixing this problem. In your

[jira] [Updated] (HDFS-8015) Erasure Coding: local and remote block writer for coding work in DataNode

2015-04-27 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8015: Attachment: HDFS-8015-000.patch Patch 000 is an initial version showing the implementation of BlockWriter used in

[jira] [Commented] (HDFS-8015) Erasure Coding: local and remote block writer for coding work in DataNode

2015-04-27 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14513671#comment-14513671 ] Li Bo commented on HDFS-8015: - hi, Yi I think there’re several ways to handle writing a decoded

[jira] [Updated] (HDFS-7786) Handle slow writers for DFSStripedOutputStream

2015-04-27 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-7786: Assignee: Keisuke Ogiwara (was: Li Bo) Handle slow writers for DFSStripedOutputStream

[jira] [Commented] (HDFS-8228) TestDFSStripedInputStream#testFileSmallerThanOneStripe2 failed

2015-04-23 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14510487#comment-14510487 ] Li Bo commented on HDFS-8228: - Thanks Jing for fixing this bug.

[jira] [Commented] (HDFS-8136) Client gets and uses EC schema when reads and writes a stripping file

2015-04-21 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504624#comment-14504624 ] Li Bo commented on HDFS-8136: - The patch also looks good to me. When I apply the patch to

[jira] [Updated] (HDFS-8136) Client gets and uses EC schema when reads and writes a stripping file

2015-04-21 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Bo updated HDFS-8136: Attachment: HDFS-8136-005.patch Client gets and uses EC schema when reads and writes a stripping file

[jira] [Commented] (HDFS-8136) Client gets and uses EC schema when reads and writes a stripping file

2015-04-21 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506461#comment-14506461 ] Li Bo commented on HDFS-8136: - HDFS-8190 has given and committed changes to

[jira] [Commented] (HDFS-8172) Failure handling: DFSStripedOutputStream continues writing with enough remaining datanodes

2015-04-20 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504075#comment-14504075 ] Li Bo commented on HDFS-8172: - hi, Nicholas Sorry for the late reply. At first I think we can

[jira] [Commented] (HDFS-8015) Erasure Coding: local and remote block writer for coding work in DataNode

2015-04-20 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504163#comment-14504163 ] Li Bo commented on HDFS-8015: - hi, Nichilas I am now working on this subtask to support erasure

[jira] [Work started] (HDFS-8171) Extend BlockSender to support multiple block data source

2015-04-17 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-8171 started by Li Bo. --- Extend BlockSender to support multiple block data source

[jira] [Created] (HDFS-8171) Extend BlockSender to support multiple block data source

2015-04-17 Thread Li Bo (JIRA)
Li Bo created HDFS-8171: --- Summary: Extend BlockSender to support multiple block data source Key: HDFS-8171 URL: https://issues.apache.org/jira/browse/HDFS-8171 Project: Hadoop HDFS Issue Type:

[jira] [Work started] (HDFS-8172) Failure handling: DFSStripedOutputStream continues writing with enough remaining datanodes

2015-04-17 Thread Li Bo (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-8172 started by Li Bo. --- Failure handling: DFSStripedOutputStream continues writing with enough remaining datanodes

[jira] [Created] (HDFS-8172) Failure handling: DFSStripedOutputStream continues writing with enough remaining datanodes

2015-04-17 Thread Li Bo (JIRA)
Li Bo created HDFS-8172: --- Summary: Failure handling: DFSStripedOutputStream continues writing with enough remaining datanodes Key: HDFS-8172 URL: https://issues.apache.org/jira/browse/HDFS-8172 Project: Hadoop

<    1   2   3   4   >