[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[
https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616673#comment-16616673
]
Hadoop QA commented on HADOOP-15558:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 128 new + 17 unchanged - 2 fixed = 145 total (was 19) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
3s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | HADOOP-15558 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930252/HADOOP-15558.002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 53e0dade4a21 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 82fbbd5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15201/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15201/testReport/ |
| Max. process+thread count | 1367 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/15201/console |
|
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[
https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616641#comment-16616641
]
Chaitanya Mukka commented on HADOOP-15558:
--
Hi [~Sammi],
Sorry for the delay. Please find the answers inline.
{quote}1. The encoding and decoding of Clay Codes involve PFT, PRT and RS
computation. So basically the idea is to reduce the network throughput and disk
read during data repair phase by adding additional computations. In single data
node failure case, Clay Codes can save 2/3 network bandwidth compared with RS.
In the worst case, Clay Codes will behave the same as RS from network bandwidth
wise. Given that most of the failures are single node failure in the storage
cluster, the cluster can benefit from Clay Codes with no doubt. I assume all
the benchmark data in slides are collected in a single data node failure case.
Correct me if it's not correct.
{quote}
Yes, that's right most of the data in slides is for single data node failure
cases. The Clay codes can handle some multiple erasures efficiently as well.
However, the current implementation is specific to a single node erasure case.
{quote}2. On P22 of slides, it says " total encoding time remains the same
while Clay Codec has 70% higher encode computation time". Confused, could you
explain it further?
{quote}
The encoding time includes the time taken to transfer and write all the blocks
to disk as well, however, the computation time is the time taken to compute the
coded blocks. The computation time of RS is small in comparison to the time
taken to send the coded block across the network and to write them to the disks.
{quote}3. On P21 of slices, Fragmented Read, it says there is no impact on SSD
when sub-chunk size reaches 4KB. Do you have any data for HDD? Since the
Hadoop/HDFS, HDD is still the majority.
{quote}
We do not currently have data for HDD. We believe that even for HDDs if the
sub-chunk size is large enough, then the contiguous data read will be of
sub-chunk size in the worst case scenario.
{quote}4. P23, what does the "Degraded I/O" scenario means in the slides?
{quote}
It means the read/write I/O speed when node recovery is happening in the
background.
{quote}5. From the slices, we can see to configure a Clay Codec, k, m, d, and
sub-trunk size all have matter. While in the implementation, only k and m are
configurable. What about d and sub-trunk?
{quote}
We have currently implemented the code for d=k+m-1 (the case where the network
bandwidth is minimum) in Hadoop patch. The Ceph implementation has it for any
k,m,d.
We plan to extend the implementation to dhttps://www.usenix.org/system/files/conference/fast18/fast18-vajha.pdf] for
details on the matrix. The example matrix given in there is just an example.
PFT and PRT can be realized by using any (4,2) MDS code as well, where the
symbols (C,C*,U,U*) are related by the (4,2) i.e. k=2 and m=2, MDS code. In
case of PFT, C, C* are assumed to be erased and are recovered from U, U*. For
PRT, U, U* are assumed to be erased and are recovered from C, C*.
{quote}7. For implementation part, is clone input blocks a must when
prepareEncodingStep? Also could you add more comments, such as which part is
PFT computation, and PRT computation. I will go through the code again later.
Also ClayCodeUtil is better to be placed in a new file.
{quote}
The sequential decoding of ClayCode requires us to store the previously decoded
layers while still maintaining the original encoded copies of the same. This
was the main reason we had to maintain a clone of the blocks.
Also the reason ClayCodeUtil was placed inside the ClayCodeDecodingStep class
due to its tight coupling with the same. This was a design decision we made due
to the relationship. We will try to reconsider and see if we can refactor it
better.
Also, we will surely we will make the necessary changes to take care of the
code style and update the patch.
> Implementation of Clay Codes plugin (Coupled Layer MSR codes)
> --
>
> Key: HADOOP-15558
> URL: https://issues.apache.org/jira/browse/HADOOP-15558
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Chaitanya Mukka
>Assignee: Chaitanya Mukka
>Priority: Major
> Attachments: ClayCodeCodecDesign-20180630.pdf,
> HADOOP-15558.001.patch, HADOOP-15558.002.patch
>
>
> [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are
> new erasure codes developed as a research project at Codes and Signal Design
> Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has
> been shown to reduce repair network traffic, disk read and repair times by
> factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same
> parameters.
> This
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[
https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588900#comment-16588900
]
Sammi Chen commented on HADOOP-15558:
-
Hi [~shreya2205], several comments after gone through the slides and code.
1. The encoding and decoding of Clay Codes involve PFT, PRT and RS computation.
So basically the idea is to reduce the network throughput and disk read during
data repair phase by adding additional computations. In single data node
failure case, Clay Codes can save 2/3 network bandwidth compared with RS. In
worst case, Clay Codes will behave the same as RS from network bandwidth wise.
Given that most of the failures are single node failure in the storage cluster,
cluster can benefit from Clay Codes with no doubt. I assume all the benchmark
data in slides are collected in single data node failure case. Correct me if
it's not correct.
2. On P22 of slides, it says " total encoding time remains the same while Clay
Codec has 70% higher encode computation time". Confused, could you explain it
further?
3. On P21 of slices, Fragmented Read, it says there is no impact on SSD when
sub-chunk size reaches 4KB. DO you have any data for HDD? Since the
Hadoop/HDFS, HDD is still the majority.
4. P23, what does the "Degraded I/O" scenario means in the slides?
5. From the slices, we can see to configure a Clay Codec, k, m, d and sub-trunk
size all have matter. While in the implementation, only k and m are
configurable. What about d and sub-trunk?
6. I googled a lot but found very few links about PFT and PRT matrix. Do you
have any documents for them?
7. For implementation part, is clone input blocks a must when
prepareEncodingStep? Also could you add more comments, such as whih part is
PFT computation, and PRT computation. I will go through the code again later.
Also ClayCodeUtil is better to be placed in a new file.
8. Code style. Here are a list of Hadoop code styles to follow.
a. Import * is not recommended
b. a line cannot exceed 80 characters
c. tab is 4 spaces
d. new line indent is 2 spaces
e. cross line indent is 4 spaces
f. remove unnecessary empty line
g. 1 space between operator and value,
for examples,
if (rsRawDecoder==null) { => if (rsRawDecoder == null) {
new ErasureCoderOptions(2,2); => new ErasureCoderOptions(2,
2);
if(erasedIndexes.length==1){ =>if (erasedIndexes.length ==
1) {
> Implementation of Clay Codes plugin (Coupled Layer MSR codes)
> --
>
> Key: HADOOP-15558
> URL: https://issues.apache.org/jira/browse/HADOOP-15558
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Chaitanya Mukka
>Assignee: Chaitanya Mukka
>Priority: Major
> Attachments: ClayCodeCodecDesign-20180630.pdf,
> HADOOP-15558.001.patch, HADOOP-15558.002.patch
>
>
> [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are
> new erasure codes developed as a research project at Codes and Signal Design
> Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has
> been shown to reduce repair network traffic, disk read and repair times by
> factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same
> parameters.
> This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable
> erasure codec.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[ https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566369#comment-16566369 ] Shreya Gupta commented on HADOOP-15558: --- Hi [~Sammi Chen], thanks for starting to look at our code. Take your time to go through the code and design documentation and please feel free to contact in case of any doubts. > Implementation of Clay Codes plugin (Coupled Layer MSR codes) > -- > > Key: HADOOP-15558 > URL: https://issues.apache.org/jira/browse/HADOOP-15558 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Chaitanya Mukka >Assignee: Chaitanya Mukka >Priority: Major > Attachments: ClayCodeCodecDesign-20180630.pdf, > HADOOP-15558.001.patch, HADOOP-15558.002.patch > > > [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are > new erasure codes developed as a research project at Codes and Signal Design > Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has > been shown to reduce repair network traffic, disk read and repair times by > factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same > parameters. > This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable > erasure codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[ https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566317#comment-16566317 ] Sammi Chen commented on HADOOP-15558: - Hi [~cmukka20] and [~shreya2205], thanks for firing this JIRA. I will take some time to go through the design document & patch, and provide feedback later. > Implementation of Clay Codes plugin (Coupled Layer MSR codes) > -- > > Key: HADOOP-15558 > URL: https://issues.apache.org/jira/browse/HADOOP-15558 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Chaitanya Mukka >Assignee: Chaitanya Mukka >Priority: Major > Attachments: ClayCodeCodecDesign-20180630.pdf, > HADOOP-15558.001.patch, HADOOP-15558.002.patch > > > [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are > new erasure codes developed as a research project at Codes and Signal Design > Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has > been shown to reduce repair network traffic, disk read and repair times by > factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same > parameters. > This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable > erasure codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[
https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16532594#comment-16532594
]
genericqa commented on HADOOP-15558:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 33m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
39s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15558 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930252/HADOOP-15558.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f0e519d02023 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 3b63715 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14859/testReport/ |
| Max. process+thread count | 1491 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14859/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Implementation of Clay Codes plugin (Coupled Layer MSR codes)
> --
>
>
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[
https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16531309#comment-16531309
]
genericqa commented on HADOOP-15558:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
44s{color} | {color:red} hadoop-common-project/hadoop-common generated 1 new +
0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 4s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}124m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | Useless object stored in variable inputPositions of method
org.apache.hadoop.io.erasurecode.coder.ClayCodeErasureDecodingStep.doDecodeSingle(ByteBuffer[][],
ByteBuffer[][], int, int, boolean) At
ClayCodeErasureDecodingStep.java:inputPositions of method
org.apache.hadoop.io.erasurecode.coder.ClayCodeErasureDecodingStep.doDecodeSingle(ByteBuffer[][],
ByteBuffer[][], int, int, boolean) At ClayCodeErasureDecodingStep.java:[line
147] |
| Failed junit tests | hadoop.io.erasurecode.codec.TestClayCodeErasureCodec |
| | hadoop.io.erasurecode.TestECSchema |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15558 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12930099/HADOOP-15558.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0a4091e2a087 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| g
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[ https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529756#comment-16529756 ] Chaitanya Mukka commented on HADOOP-15558: -- Hi. We have uploaded both the design doc and the patch. We would be grateful if they are reviewed and any improvements suggested. > Implementation of Clay Codes plugin (Coupled Layer MSR codes) > -- > > Key: HADOOP-15558 > URL: https://issues.apache.org/jira/browse/HADOOP-15558 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Chaitanya Mukka >Assignee: Chaitanya Mukka >Priority: Major > Attachments: ClayCodeCodecDesign-20180630.pdf, HADOOP-15558.001.patch > > > [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are > new erasure codes developed as a research project at Codes and Signal Design > Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has > been shown to reduce repair network traffic, disk read and repair times by > factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same > parameters. > This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable > erasure codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[ https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16523573#comment-16523573 ] Shreya Gupta commented on HADOOP-15558: --- Hi. Meanwhile one can have a look at these [slides|https://www.usenix.org/sites/default/files/conference/protected-files/fast18_slides_vajha.pdf] to better understand how Clay Codes work. > Implementation of Clay Codes plugin (Coupled Layer MSR codes) > -- > > Key: HADOOP-15558 > URL: https://issues.apache.org/jira/browse/HADOOP-15558 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Chaitanya Mukka >Assignee: Chaitanya Mukka >Priority: Major > > [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are > new erasure codes developed as a research project at Codes and Signal Design > Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has > been shown to reduce repair network traffic, disk read and repair times by > factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same > parameters. > This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable > erasure codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[ https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16523244#comment-16523244 ] Chaitanya Mukka commented on HADOOP-15558: -- Hey [~jojochuang]. Thanks a lot. We will do the necessary as soon as possible. Also, we are new to Jira and Hadoop and we would appreciate if you could point us on how to submit patches/pull requests and how the QA integration works. > Implementation of Clay Codes plugin (Coupled Layer MSR codes) > -- > > Key: HADOOP-15558 > URL: https://issues.apache.org/jira/browse/HADOOP-15558 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Chaitanya Mukka >Assignee: Chaitanya Mukka >Priority: Major > > [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are > new erasure codes developed as a research project at Codes and Signal Design > Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has > been shown to reduce repair network traffic, disk read and repair times by > factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same > parameters. > This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable > erasure codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[ https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522521#comment-16522521 ] Wei-Chiu Chuang commented on HADOOP-15558: -- Hi [~cmukka20], [~shreya2205] thanks for filing the jira. I've added both of you to JIRA's contributor list, so you'll be able to assign jiras to you. When you get a chance, please post a design doc or a description of the new codec. Thanks! > Implementation of Clay Codes plugin (Coupled Layer MSR codes) > -- > > Key: HADOOP-15558 > URL: https://issues.apache.org/jira/browse/HADOOP-15558 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Chaitanya Mukka >Priority: Major > > [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are > new erasure codes developed as a research project at Codes and Signal Design > Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has > been shown to reduce repair network traffic, disk read and repair times by > factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same > parameters. > This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable > erasure codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15558) Implementation of Clay Codes plugin (Coupled Layer MSR codes)
[ https://issues.apache.org/jira/browse/HADOOP-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16522177#comment-16522177 ] Chaitanya Mukka commented on HADOOP-15558: -- Hi. Me and [~shreya2205] are interns at CSD Lab, IISc and will be working to implement this. > Implementation of Clay Codes plugin (Coupled Layer MSR codes) > -- > > Key: HADOOP-15558 > URL: https://issues.apache.org/jira/browse/HADOOP-15558 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Chaitanya Mukka >Priority: Major > > [Clay Codes|https://www.usenix.org/conference/fast18/presentation/vajha] are > new erasure codes developed as a research project at Codes and Signal Design > Lab, IISc Bangalore. A particular Clay code, with storage overhead 1.25x, has > been shown to reduce repair network traffic, disk read and repair times by > factors of 2.9, 3.4 and 3 respectively compared to the RS codes with the same > parameters. > This Jira aims to introduce Clay Codes to HDFS-EC as one of the pluggable > erasure codec. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
