[jira] [Updated] (HDFS-1783) Ability for HDFS client to write replicas in parallel
[ https://issues.apache.org/jira/browse/HDFS-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-1783: --- Labels: BB2015-05-TBR (was: ) Ability for HDFS client to write replicas in parallel - Key: HDFS-1783 URL: https://issues.apache.org/jira/browse/HDFS-1783 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Reporter: dhruba borthakur Assignee: Rosa Ali Labels: BB2015-05-TBR Attachments: HDFS-1783-trunk-v2.patch, HDFS-1783-trunk-v3.patch, HDFS-1783-trunk-v4.patch, HDFS-1783-trunk-v5.patch, HDFS-1783-trunk.patch The current implementation of HDFS pipelines the writes to the three replicas. This introduces some latency for realtime latency sensitive applications. An alternate implementation that allows the client to write all replicas in parallel gives much better response times to these applications. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-1783) Ability for HDFS client to write replicas in parallel
[ https://issues.apache.org/jira/browse/HDFS-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HDFS-1783: Assignee: (was: Lars Hofhansl) Ability for HDFS client to write replicas in parallel - Key: HDFS-1783 URL: https://issues.apache.org/jira/browse/HDFS-1783 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Reporter: dhruba borthakur Attachments: HDFS-1783-trunk.patch, HDFS-1783-trunk-v2.patch, HDFS-1783-trunk-v3.patch, HDFS-1783-trunk-v4.patch, HDFS-1783-trunk-v5.patch The current implementation of HDFS pipelines the writes to the three replicas. This introduces some latency for realtime latency sensitive applications. An alternate implementation that allows the client to write all replicas in parallel gives much better response times to these applications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1783) Ability for HDFS client to write replicas in parallel
[ https://issues.apache.org/jira/browse/HDFS-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HDFS-1783: Attachment: HDFS-1783-trunk-v5.patch Also adds a subclass of TestPipelinesFailover running all tests with PARALLEL_WRITES. FileSystem.append itself does not support parallel writes (as of this patch). I am generally not quite clear what the difference between FileSystem.append and FileSystem.create(..., CreateFlag.APPEND, ...) is supposed to be. Ability for HDFS client to write replicas in parallel - Key: HDFS-1783 URL: https://issues.apache.org/jira/browse/HDFS-1783 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Reporter: dhruba borthakur Assignee: Lars Hofhansl Attachments: HDFS-1783-trunk-v2.patch, HDFS-1783-trunk-v3.patch, HDFS-1783-trunk-v4.patch, HDFS-1783-trunk-v5.patch, HDFS-1783-trunk.patch The current implementation of HDFS pipelines the writes to the three replicas. This introduces some latency for realtime latency sensitive applications. An alternate implementation that allows the client to write all replicas in parallel gives much better response times to these applications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1783) Ability for HDFS client to write replicas in parallel
[ https://issues.apache.org/jira/browse/HDFS-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HDFS-1783: Attachment: HDFS-1783-trunk-v4.patch * added Stack's suggestions (thanks Stack): * added subclasses of TestReplication and TestDatanodeDeath, that run all tests with parallel writes enabled * found a problem with error handling (with TestDatanodeDeathWithParallelWrites), fixed it * renamed s[] to sockets[] Ability for HDFS client to write replicas in parallel - Key: HDFS-1783 URL: https://issues.apache.org/jira/browse/HDFS-1783 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Reporter: dhruba borthakur Assignee: Lars Hofhansl Attachments: HDFS-1783-trunk-v2.patch, HDFS-1783-trunk-v3.patch, HDFS-1783-trunk-v4.patch, HDFS-1783-trunk.patch The current implementation of HDFS pipelines the writes to the three replicas. This introduces some latency for realtime latency sensitive applications. An alternate implementation that allows the client to write all replicas in parallel gives much better response times to these applications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1783) Ability for HDFS client to write replicas in parallel
[ https://issues.apache.org/jira/browse/HDFS-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HDFS-1783: Attachment: HDFS-1783-trunk-v3.patch Added a new DataNodeMetric: ChunksVerified. Since with parallel writes each DN has to verify each chunk rather than just the last DN in a pipeline, this is an interesting metric to capture. Also added a simple test based on that metric. Ability for HDFS client to write replicas in parallel - Key: HDFS-1783 URL: https://issues.apache.org/jira/browse/HDFS-1783 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Reporter: dhruba borthakur Assignee: Lars Hofhansl Attachments: HDFS-1783-trunk-v2.patch, HDFS-1783-trunk-v3.patch, HDFS-1783-trunk.patch The current implementation of HDFS pipelines the writes to the three replicas. This introduces some latency for realtime latency sensitive applications. An alternate implementation that allows the client to write all replicas in parallel gives much better response times to these applications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1783) Ability for HDFS client to write replicas in parallel
[ https://issues.apache.org/jira/browse/HDFS-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HDFS-1783: Status: Patch Available (was: Open) Let's see how many tests this breaks. Ability for HDFS client to write replicas in parallel - Key: HDFS-1783 URL: https://issues.apache.org/jira/browse/HDFS-1783 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Reporter: dhruba borthakur Assignee: Lars Hofhansl Attachments: HDFS-1783-trunk-v2.patch, HDFS-1783-trunk.patch The current implementation of HDFS pipelines the writes to the three replicas. This introduces some latency for realtime latency sensitive applications. An alternate implementation that allows the client to write all replicas in parallel gives much better response times to these applications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1783) Ability for HDFS client to write replicas in parallel
[ https://issues.apache.org/jira/browse/HDFS-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HDFS-1783: Attachment: HDFS-1783-trunk-v2.patch Patch rebased for HDFS-744. * Added a first simple test * Fixed a bunch of problems I had introduced when I ported the patch over Ability for HDFS client to write replicas in parallel - Key: HDFS-1783 URL: https://issues.apache.org/jira/browse/HDFS-1783 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Reporter: dhruba borthakur Assignee: Lars Hofhansl Attachments: HDFS-1783-trunk-v2.patch, HDFS-1783-trunk.patch The current implementation of HDFS pipelines the writes to the three replicas. This introduces some latency for realtime latency sensitive applications. An alternate implementation that allows the client to write all replicas in parallel gives much better response times to these applications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1783) Ability for HDFS client to write replicas in parallel
[ https://issues.apache.org/jira/browse/HDFS-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HDFS-1783: Attachment: HDFS-1783-trunk.patch Sketch of a patch (mostly just an adaption of Dhrubas change). Only tested whether it compiles so far. Will do some testing and add unit tests soon. Using this may eat your first born so use at your own risk. Ability for HDFS client to write replicas in parallel - Key: HDFS-1783 URL: https://issues.apache.org/jira/browse/HDFS-1783 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Reporter: dhruba borthakur Assignee: Lars Hofhansl Attachments: HDFS-1783-trunk.patch The current implementation of HDFS pipelines the writes to the three replicas. This introduces some latency for realtime latency sensitive applications. An alternate implementation that allows the client to write all replicas in parallel gives much better response times to these applications. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
