[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937939#comment-14937939
]
Hudson commented on HDFS-8203:
--
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #473 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/473/])
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
(zhezhang: rev a17cedb44c2a98d863407fab95c2b7f0893d0727)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
> Erasure Coding: Seek and other Ops in DFSStripedInputStream.
>
>
> Key: HDFS-8203
> URL: https://issues.apache.org/jira/browse/HDFS-8203
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
> HDFS-8203.003.patch
>
>
> In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
> {{DFSStripedInputStream}}, we also need handle other operations, such as
> {{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938159#comment-14938159
]
Hudson commented on HDFS-8203:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #439 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/439/])
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
(zhezhang: rev a17cedb44c2a98d863407fab95c2b7f0893d0727)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
> Erasure Coding: Seek and other Ops in DFSStripedInputStream.
>
>
> Key: HDFS-8203
> URL: https://issues.apache.org/jira/browse/HDFS-8203
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
> HDFS-8203.003.patch
>
>
> In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
> {{DFSStripedInputStream}}, we also need handle other operations, such as
> {{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938728#comment-14938728
]
Hudson commented on HDFS-8203:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #465 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/465/])
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
(zhezhang: rev a17cedb44c2a98d863407fab95c2b7f0893d0727)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: Seek and other Ops in DFSStripedInputStream.
>
>
> Key: HDFS-8203
> URL: https://issues.apache.org/jira/browse/HDFS-8203
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
> HDFS-8203.003.patch
>
>
> In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
> {{DFSStripedInputStream}}, we also need handle other operations, such as
> {{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938537#comment-14938537
]
Hudson commented on HDFS-8203:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #2408 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2408/])
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
(zhezhang: rev a17cedb44c2a98d863407fab95c2b7f0893d0727)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: Seek and other Ops in DFSStripedInputStream.
>
>
> Key: HDFS-8203
> URL: https://issues.apache.org/jira/browse/HDFS-8203
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
> HDFS-8203.003.patch
>
>
> In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
> {{DFSStripedInputStream}}, we also need handle other operations, such as
> {{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937686#comment-14937686
]
Hudson commented on HDFS-8203:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #1203 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/1203/])
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
(zhezhang: rev a17cedb44c2a98d863407fab95c2b7f0893d0727)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
> Erasure Coding: Seek and other Ops in DFSStripedInputStream.
>
>
> Key: HDFS-8203
> URL: https://issues.apache.org/jira/browse/HDFS-8203
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
> HDFS-8203.003.patch
>
>
> In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
> {{DFSStripedInputStream}}, we also need handle other operations, such as
> {{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937240#comment-14937240
]
Hudson commented on HDFS-8203:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #2379 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/2379/])
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
(zhezhang: rev a17cedb44c2a98d863407fab95c2b7f0893d0727)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: Seek and other Ops in DFSStripedInputStream.
>
>
> Key: HDFS-8203
> URL: https://issues.apache.org/jira/browse/HDFS-8203
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
> HDFS-8203.003.patch
>
>
> In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
> {{DFSStripedInputStream}}, we also need handle other operations, such as
> {{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14937436#comment-14937436
]
Hudson commented on HDFS-8203:
--
FAILURE: Integrated in Hadoop-trunk-Commit #8548 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8548/])
HDFS-8203. Erasure Coding: Seek and other Ops in DFSStripedInputStream.
(zhezhang: rev a17cedb44c2a98d863407fab95c2b7f0893d0727)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestWriteReadStripedFile.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-EC-7285.txt
> Erasure Coding: Seek and other Ops in DFSStripedInputStream.
>
>
> Key: HDFS-8203
> URL: https://issues.apache.org/jira/browse/HDFS-8203
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Yi Liu
>Assignee: Yi Liu
> Fix For: HDFS-7285
>
> Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
> HDFS-8203.003.patch
>
>
> In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
> {{DFSStripedInputStream}}, we also need handle other operations, such as
> {{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537472#comment-14537472
]
Yi Liu commented on HDFS-8203:
--
Jing, I just found we should also override {{seekToNewSource}}, ideally, for
striped block, an internal block is only on one datanode, so
DFSStripedInputStream#seekToNewSource should return false, I opened HDFS-8363
which also removes a hardcode for {{groupSize}}.
Erasure Coding: Seek and other Ops in DFSStripedInputStream.
Key: HDFS-8203
URL: https://issues.apache.org/jira/browse/HDFS-8203
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: HDFS-7285
Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
HDFS-8203.003.patch
In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
{{DFSStripedInputStream}}, we also need handle other operations, such as
{{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533104#comment-14533104
]
Jing Zhao commented on HDFS-8203:
-
Thanks for updating the patch, Yi! The 003 patch looks pretty good to me. +1.
There are two unnecessary int conversion in the code. I will remove them while
committing the patch.
Erasure Coding: Seek and other Ops in DFSStripedInputStream.
Key: HDFS-8203
URL: https://issues.apache.org/jira/browse/HDFS-8203
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
HDFS-8203.003.patch
In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
{{DFSStripedInputStream}}, we also need handle other operations, such as
{{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533688#comment-14533688
]
Yi Liu commented on HDFS-8203:
--
Thanks a lot for the review and commit, Jing! :)
Erasure Coding: Seek and other Ops in DFSStripedInputStream.
Key: HDFS-8203
URL: https://issues.apache.org/jira/browse/HDFS-8203
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Fix For: HDFS-7285
Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch,
HDFS-8203.003.patch
In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
{{DFSStripedInputStream}}, we also need handle other operations, such as
{{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531770#comment-14531770
]
Jing Zhao commented on HDFS-8203:
-
Thanks Yi! The patch looks pretty good to me. Some minor comments:
# In the seek function, do we also want to do a special handling for the
scenario that the targetPos is before pos but still locates inside of the
curStripeRange?
# Nitty-pick: we can rename the local variable pos to something else in order
to avoid the confusion with {{DFSInputStream#pos}}.
{code}
for (int i = startCell; i numCell; i++) {
int pos = i == startCell ? bufOffset : cellSize * i;
curStripeBuf.position(pos);
{code}
# Maybe we can also add some simple tests to cover cases seeking to
cell/stripe/block boundaries?
Erasure Coding: Seek and other Ops in DFSStripedInputStream.
Key: HDFS-8203
URL: https://issues.apache.org/jira/browse/HDFS-8203
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8203.001.patch, HDFS-8203.002.patch
In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
{{DFSStripedInputStream}}, we also need handle other operations, such as
{{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14528406#comment-14528406
]
Yi Liu commented on HDFS-8203:
--
Thanks Jing for the comment, you are right, it will be more simpler, I will
update the patch with test later.
Erasure Coding: Seek and other Ops in DFSStripedInputStream.
Key: HDFS-8203
URL: https://issues.apache.org/jira/browse/HDFS-8203
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8203.001.patch
In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
{{DFSStripedInputStream}}, we also need handle other operations, such as
{{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8203) Erasure Coding: Seek and other Ops in DFSStripedInputStream.
[
https://issues.apache.org/jira/browse/HDFS-8203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527305#comment-14527305
]
Jing Zhao commented on HDFS-8203:
-
Thanks for working on this, Yi! Currently we're trying to provide a stripe
buffer in HDFS-8281. If that gets in we may need to update the seek
implementation accordingly (it may get simpler).
Erasure Coding: Seek and other Ops in DFSStripedInputStream.
Key: HDFS-8203
URL: https://issues.apache.org/jira/browse/HDFS-8203
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-8203.001.patch
In HDFS-7782 and HDFS-8033, we handle pread and stateful read for
{{DFSStripedInputStream}}, we also need handle other operations, such as
{{seek}}, zerocopy read ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
