[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

2015-09-30 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938635#comment-14938635
 ] 

Hudson commented on HDFS-7672:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/])
HDFS-7672. Handle write failure for stripping blocks and refactor the 
(zhezhang: rev 220ca960bce970d5969b9af570a3ce43360b7e2b)
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java


> Erasure Coding: consolidate streamer coordination logic and handle failure 
> when writing striped blocks
> --
>
> Key: HDFS-7672
> URL: https://issues.apache.org/jira/browse/HDFS-7672
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h7672_20150504.patch, h7672_20150504b.patch, 
> h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch
>
>
> In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data 
> blocks and 3 parity blocks concurrently.  We need to handle datanode or 
> network failures when writing a EC BlockGroup.
> We also refactor the existing code in DFSStripedOutputStream and 
> StripedDataStreamer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

2015-09-30 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938443#comment-14938443
 ] 

Hudson commented on HDFS-7672:
--

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/])
HDFS-7672. Handle write failure for stripping blocks and refactor the 
(zhezhang: rev 220ca960bce970d5969b9af570a3ce43360b7e2b)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java


> Erasure Coding: consolidate streamer coordination logic and handle failure 
> when writing striped blocks
> --
>
> Key: HDFS-7672
> URL: https://issues.apache.org/jira/browse/HDFS-7672
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h7672_20150504.patch, h7672_20150504b.patch, 
> h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch
>
>
> In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data 
> blocks and 3 parity blocks concurrently.  We need to handle datanode or 
> network failures when writing a EC BlockGroup.
> We also refactor the existing code in DFSStripedOutputStream and 
> StripedDataStreamer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

2015-09-30 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14938065#comment-14938065
 ] 

Hudson commented on HDFS-7672:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-7672. Handle write failure for stripping blocks and refactor the 
(zhezhang: rev 220ca960bce970d5969b9af570a3ce43360b7e2b)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt


> Erasure Coding: consolidate streamer coordination logic and handle failure 
> when writing striped blocks
> --
>
> Key: HDFS-7672
> URL: https://issues.apache.org/jira/browse/HDFS-7672
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h7672_20150504.patch, h7672_20150504b.patch, 
> h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch
>
>
> In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data 
> blocks and 3 parity blocks concurrently.  We need to handle datanode or 
> network failures when writing a EC BlockGroup.
> We also refactor the existing code in DFSStripedOutputStream and 
> StripedDataStreamer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

2015-09-30 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937846#comment-14937846
 ] 

Hudson commented on HDFS-7672:
--

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-7672. Handle write failure for stripping blocks and refactor the 
(zhezhang: rev 220ca960bce970d5969b9af570a3ce43360b7e2b)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java


> Erasure Coding: consolidate streamer coordination logic and handle failure 
> when writing striped blocks
> --
>
> Key: HDFS-7672
> URL: https://issues.apache.org/jira/browse/HDFS-7672
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h7672_20150504.patch, h7672_20150504b.patch, 
> h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch
>
>
> In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data 
> blocks and 3 parity blocks concurrently.  We need to handle datanode or 
> network failures when writing a EC BlockGroup.
> We also refactor the existing code in DFSStripedOutputStream and 
> StripedDataStreamer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

2015-09-30 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937589#comment-14937589
 ] 

Hudson commented on HDFS-7672:
--

FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/])
HDFS-7672. Handle write failure for stripping blocks and refactor the 
(zhezhang: rev 220ca960bce970d5969b9af570a3ce43360b7e2b)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java


> Erasure Coding: consolidate streamer coordination logic and handle failure 
> when writing striped blocks
> --
>
> Key: HDFS-7672
> URL: https://issues.apache.org/jira/browse/HDFS-7672
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h7672_20150504.patch, h7672_20150504b.patch, 
> h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch
>
>
> In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data 
> blocks and 3 parity blocks concurrently.  We need to handle datanode or 
> network failures when writing a EC BlockGroup.
> We also refactor the existing code in DFSStripedOutputStream and 
> StripedDataStreamer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

2015-09-30 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937342#comment-14937342
 ] 

Hudson commented on HDFS-7672:
--

FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/8548/])
HDFS-7672. Handle write failure for stripping blocks and refactor the 
(zhezhang: rev 220ca960bce970d5969b9af570a3ce43360b7e2b)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java


> Erasure Coding: consolidate streamer coordination logic and handle failure 
> when writing striped blocks
> --
>
> Key: HDFS-7672
> URL: https://issues.apache.org/jira/browse/HDFS-7672
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h7672_20150504.patch, h7672_20150504b.patch, 
> h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch
>
>
> In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data 
> blocks and 3 parity blocks concurrently.  We need to handle datanode or 
> network failures when writing a EC BlockGroup.
> We also refactor the existing code in DFSStripedOutputStream and 
> StripedDataStreamer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

2015-09-30 Thread Hudson (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14937145#comment-14937145
 ] 

Hudson commented on HDFS-7672:
--

FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/])
HDFS-7672. Handle write failure for stripping blocks and refactor the 
(zhezhang: rev 220ca960bce970d5969b9af570a3ce43360b7e2b)
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java
* 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java


> Erasure Coding: consolidate streamer coordination logic and handle failure 
> when writing striped blocks
> --
>
> Key: HDFS-7672
> URL: https://issues.apache.org/jira/browse/HDFS-7672
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h7672_20150504.patch, h7672_20150504b.patch, 
> h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch
>
>
> In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data 
> blocks and 3 parity blocks concurrently.  We need to handle datanode or 
> network failures when writing a EC BlockGroup.
> We also refactor the existing code in DFSStripedOutputStream and 
> StripedDataStreamer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

2015-05-06 Thread Tsz Wo Nicholas Sze (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531724#comment-14531724
 ] 

Tsz Wo Nicholas Sze commented on HDFS-7672:
---

They will be aligned other, it won't get the next located block.

> Erasure Coding: consolidate streamer coordination logic and handle failure 
> when writing striped blocks
> --
>
> Key: HDFS-7672
> URL: https://issues.apache.org/jira/browse/HDFS-7672
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h7672_20150504.patch, h7672_20150504b.patch, 
> h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch
>
>
> In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data 
> blocks and 3 parity blocks concurrently.  We need to handle datanode or 
> network failures when writing a EC BlockGroup.
> We also refactor the existing code in DFSStripedOutputStream and 
> StripedDataStreamer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

2015-05-06 Thread Zhe Zhang (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531708#comment-14531708
 ] 

Zhe Zhang commented on HDFS-7672:
-

Thanks Nicholas. Actually it's also weird to maintain a list of {{endBlock}}. 
With fast and slow writers we are not guaranteed that the head {{endBlock}} of 
each streamer's queue are aligned on the same block group.

> Erasure Coding: consolidate streamer coordination logic and handle failure 
> when writing striped blocks
> --
>
> Key: HDFS-7672
> URL: https://issues.apache.org/jira/browse/HDFS-7672
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h7672_20150504.patch, h7672_20150504b.patch, 
> h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch
>
>
> In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data 
> blocks and 3 parity blocks concurrently.  We need to handle datanode or 
> network failures when writing a EC BlockGroup.
> We also refactor the existing code in DFSStripedOutputStream and 
> StripedDataStreamer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HDFS-7672) Erasure Coding: consolidate streamer coordination logic and handle failure when writing striped blocks

2015-05-06 Thread Tsz Wo Nicholas Sze (JIRA)

[ 
https://issues.apache.org/jira/browse/HDFS-7672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531700#comment-14531700
 ] 

Tsz Wo Nicholas Sze commented on HDFS-7672:
---

> shouldLocateFollowingBlock doesn't look good but since it's inherited I'll 
> file another JIRA to address it.

I found that we actually could check if block==null but not need another 
variable.  I will fix it in HDFS-8323.

> Erasure Coding: consolidate streamer coordination logic and handle failure 
> when writing striped blocks
> --
>
> Key: HDFS-7672
> URL: https://issues.apache.org/jira/browse/HDFS-7672
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
> Fix For: HDFS-7285
>
> Attachments: h7672_20150504.patch, h7672_20150504b.patch, 
> h7672_20150504c.patch, h7672_20150505.patch, h7672_20150505b.patch
>
>
> In *stripping* case, for (6, 3)-Reed-Solomon, a client writes to 6 data 
> blocks and 3 parity blocks concurrently.  We need to handle datanode or 
> network failures when writing a EC BlockGroup.
> We also refactor the existing code in DFSStripedOutputStream and 
> StripedDataStreamer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)