[jira] [Updated] (HDFS-10800) [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in other storages than what NN is expecting.
[ https://issues.apache.org/jira/browse/HDFS-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-10800: --- Attachment: HDFS-10800-HDFS-10285-04.patch Thanks a lot [~rakeshr], for the thoughtful suggestions. I have incorporated all of them in this patch. Please review. > [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks > which were placed in other storages than what NN is expecting. > --- > > Key: HDFS-10800 > URL: https://issues.apache.org/jira/browse/HDFS-10800 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10800-HDFS-10285-00.patch, > HDFS-10800-HDFS-10285-01.patch, HDFS-10800-HDFS-10285-02.patch, > HDFS-10800-HDFS-10285-03.patch, HDFS-10800-HDFS-10285-04.patch > > > This JIRA is for implementing a daemon thread called StoragePolicySatisfier > in nematode, which should scan the asked files blocks which were placed in > wrong storages in DNs. > The idea is: > # When user called on some files/dirs for satisfyStorage policy, They > should have tracked in NN and then StoragePolicyDaemon thread will pick one > by one file and then check the blocks which might have placed in wrong > storage in DN than what NN is expecting it to. > # After checking all, it should also construct the data structures for > the required information to move a block from one storage to another. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10800) [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in other storages than what NN is expecting.
[ https://issues.apache.org/jira/browse/HDFS-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-10800: --- Attachment: HDFS-10800-HDFS-10285-03.patch Thank you [~rakeshr] for the review. I have updated the patch fixing all your comments. for #4) I have left as is. We can refine later after adding dynamic start/stop option. TODO was there about the same. Thanks Please review! > [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks > which were placed in other storages than what NN is expecting. > --- > > Key: HDFS-10800 > URL: https://issues.apache.org/jira/browse/HDFS-10800 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10800-HDFS-10285-00.patch, > HDFS-10800-HDFS-10285-01.patch, HDFS-10800-HDFS-10285-02.patch, > HDFS-10800-HDFS-10285-03.patch > > > This JIRA is for implementing a daemon thread called StoragePolicySatisfier > in nematode, which should scan the asked files blocks which were placed in > wrong storages in DNs. > The idea is: > # When user called on some files/dirs for satisfyStorage policy, They > should have tracked in NN and then StoragePolicyDaemon thread will pick one > by one file and then check the blocks which might have placed in wrong > storage in DN than what NN is expecting it to. > # After checking all, it should also construct the data structures for > the required information to move a block from one storage to another. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10800) [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in other storages than what NN is expecting.
[ https://issues.apache.org/jira/browse/HDFS-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-10800: --- Attachment: HDFS-10800-HDFS-10285-02.patch Attached a revised new patch. I also took chance to fix a random test failure case in HDFS-10794. Please review the patch. > [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks > which were placed in other storages than what NN is expecting. > --- > > Key: HDFS-10800 > URL: https://issues.apache.org/jira/browse/HDFS-10800 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10800-HDFS-10285-00.patch, > HDFS-10800-HDFS-10285-01.patch, HDFS-10800-HDFS-10285-02.patch > > > This JIRA is for implementing a daemon thread called StoragePolicySatisfier > in nematode, which should scan the asked files blocks which were placed in > wrong storages in DNs. > The idea is: > # When user called on some files/dirs for satisfyStorage policy, They > should have tracked in NN and then StoragePolicyDaemon thread will pick one > by one file and then check the blocks which might have placed in wrong > storage in DN than what NN is expecting it to. > # After checking all, it should also construct the data structures for > the required information to move a block from one storage to another. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10800) [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in other storages than what NN is expecting.
[ https://issues.apache.org/jira/browse/HDFS-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-10800: --- Attachment: HDFS-10800-HDFS-10285-01.patch Tests were passed in my local eclipse. Since my local eclipse does not have java assertion enabled, this was not caught there. I have just updated to correct it now. > [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks > which were placed in other storages than what NN is expecting. > --- > > Key: HDFS-10800 > URL: https://issues.apache.org/jira/browse/HDFS-10800 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10800-HDFS-10285-00.patch, > HDFS-10800-HDFS-10285-01.patch > > > This JIRA is for implementing a daemon thread called StoragePolicySatisfier > in nematode, which should scan the asked files blocks which were placed in > wrong storages in DNs. > The idea is: > # When user called on some files/dirs for satisfyStorage policy, They > should have tracked in NN and then StoragePolicyDaemon thread will pick one > by one file and then check the blocks which might have placed in wrong > storage in DN than what NN is expecting it to. > # After checking all, it should also construct the data structures for > the required information to move a block from one storage to another. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10800) [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in other storages than what NN is expecting.
[ https://issues.apache.org/jira/browse/HDFS-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-10800: --- Status: Patch Available (was: Open) > [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks > which were placed in other storages than what NN is expecting. > --- > > Key: HDFS-10800 > URL: https://issues.apache.org/jira/browse/HDFS-10800 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10800-HDFS-10285-00.patch > > > This JIRA is for implementing a daemon thread called StoragePolicySatisfier > in nematode, which should scan the asked files blocks which were placed in > wrong storages in DNs. > The idea is: > # When user called on some files/dirs for satisfyStorage policy, They > should have tracked in NN and then StoragePolicyDaemon thread will pick one > by one file and then check the blocks which might have placed in wrong > storage in DN than what NN is expecting it to. > # After checking all, it should also construct the data structures for > the required information to move a block from one storage to another. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10800) [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in other storages than what NN is expecting.
[ https://issues.apache.org/jira/browse/HDFS-10800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-10800: --- Summary: [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in other storages than what NN is expecting. (was: [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks which were placed in wrong storages than what NN is expecting.) > [SPS]: Storage Policy Satisfier daemon thread in Namenode to find the blocks > which were placed in other storages than what NN is expecting. > --- > > Key: HDFS-10800 > URL: https://issues.apache.org/jira/browse/HDFS-10800 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10800-HDFS-10285-00.patch > > > This JIRA is for implementing a daemon thread called StoragePolicySatisfier > in nematode, which should scan the asked files blocks which were placed in > wrong storages in DNs. > The idea is: > # When user called on some files/dirs for satisfyStorage policy, They > should have tracked in NN and then StoragePolicyDaemon thread will pick one > by one file and then check the blocks which might have placed in wrong > storage in DN than what NN is expecting it to. > # After checking all, it should also construct the data structures for > the required information to move a block from one storage to another. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
