[jira] [Updated] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[ https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12412: --- Labels: hdfs-ec-3.0-nice-to-have (was: ) > Remove ErasureCodingWorker.stripedReadPool > -- > > Key: HDFS-12412 > URL: https://issues.apache.org/jira/browse/HDFS-12412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Labels: hdfs-ec-3.0-nice-to-have > > In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule > the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads > in each recovery task. We only need one of them to throttle the speed of > recovery process, because each EC recovery task has a fix number of source > readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the > speed of EC recovery can be throttled by {{strippedReconstructionPool}} with > {{xmitsInProgress}}. > Moreover, keeping {{stripedReadPool}} makes customer difficult to understand > and calculate the right balance between > {{dfs.datanode.ec.reconstruction.stripedread.threads}}, > {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and > {{maxReplicationStreams}}. For example, a small {{stripread.threads}} > (comparing to which {{reconstruction.threads.size}} implies), will > unnecessarily limit the speed of recovery, which leads to larger MTTR. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[ https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-12412: - Attachment: HDFS-12412.00.patch Change {{stripedReadPool}} to an unbound {{cacheThreadPool}}. [~andrew.wang], [~drankye] could you give a review? Thanks. > Remove ErasureCodingWorker.stripedReadPool > -- > > Key: HDFS-12412 > URL: https://issues.apache.org/jira/browse/HDFS-12412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12412.00.patch > > > In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule > the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads > in each recovery task. We only need one of them to throttle the speed of > recovery process, because each EC recovery task has a fix number of source > readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the > speed of EC recovery can be throttled by {{strippedReconstructionPool}} with > {{xmitsInProgress}}. > Moreover, keeping {{stripedReadPool}} makes customer difficult to understand > and calculate the right balance between > {{dfs.datanode.ec.reconstruction.stripedread.threads}}, > {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and > {{maxReplicationStreams}}. For example, a small {{stripread.threads}} > (comparing to which {{reconstruction.threads.size}} implies), will > unnecessarily limit the speed of recovery, which leads to larger MTTR. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[ https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-12412: - Status: Patch Available (was: Open) > Remove ErasureCodingWorker.stripedReadPool > -- > > Key: HDFS-12412 > URL: https://issues.apache.org/jira/browse/HDFS-12412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12412.00.patch > > > In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule > the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads > in each recovery task. We only need one of them to throttle the speed of > recovery process, because each EC recovery task has a fix number of source > readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the > speed of EC recovery can be throttled by {{strippedReconstructionPool}} with > {{xmitsInProgress}}. > Moreover, keeping {{stripedReadPool}} makes customer difficult to understand > and calculate the right balance between > {{dfs.datanode.ec.reconstruction.stripedread.threads}}, > {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and > {{maxReplicationStreams}}. For example, a small {{stripread.threads}} > (comparing to which {{reconstruction.threads.size}} implies), will > unnecessarily limit the speed of recovery, which leads to larger MTTR. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12412) Remove ErasureCodingWorker.stripedReadPool
[ https://issues.apache.org/jira/browse/HDFS-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-12412: - Attachment: HDFS-12412.01.patch Thanks for the inputs, [~andrew.wang]. Removed references of {{dfs.datanode.ec.reconstruction.stripedread.threads}} in the {01} patch. > Remove ErasureCodingWorker.stripedReadPool > -- > > Key: HDFS-12412 > URL: https://issues.apache.org/jira/browse/HDFS-12412 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12412.00.patch, HDFS-12412.01.patch > > > In {{ErasureCodingWorker}}, it uses {{stripedReconstructionPool}} to schedule > the EC recovery tasks, while uses {{stripedReadPool}} for the reader threads > in each recovery task. We only need one of them to throttle the speed of > recovery process, because each EC recovery task has a fix number of source > readers (i.e., 3 for RS(3,2)). And because of the findings in HDFS-12044, the > speed of EC recovery can be throttled by {{strippedReconstructionPool}} with > {{xmitsInProgress}}. > Moreover, keeping {{stripedReadPool}} makes customer difficult to understand > and calculate the right balance between > {{dfs.datanode.ec.reconstruction.stripedread.threads}}, > {{dfs.datanode.ec.reconstruction.stripedblock.threads.size}} and > {{maxReplicationStreams}}. For example, a small {{stripread.threads}} > (comparing to which {{reconstruction.threads.size}} implies), will > unnecessarily limit the speed of recovery, which leads to larger MTTR. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org