[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HDFS-9807:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha3
Status: Resolved (was: Patch Available)
I committed this. Thanks Ewan
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Fix For: 3.0.0-alpha3
>
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch,
> HDFS-9807.009.patch, HDFS-9807.010.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-9807:
-
Status: Patch Available (was: Open)
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch,
> HDFS-9807.009.patch, HDFS-9807.010.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-9807:
-
Attachment: HDFS-9807.010.patch
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch,
> HDFS-9807.009.patch, HDFS-9807.010.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-9807:
-
Status: Open (was: Patch Available)
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch,
> HDFS-9807.009.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-9807:
-
Attachment: HDFS-9807.009.patch
Posting a modified patch fixing some of the checkstyle errors.
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch,
> HDFS-9807.009.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-9807:
-
Status: Patch Available (was: Open)
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch,
> HDFS-9807.009.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-9807:
-
Status: Open (was: Patch Available)
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch,
> HDFS-9807.009.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-9807:
-
Status: Open (was: Patch Available)
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-9807:
-
Attachment: HDFS-9807.008.patch
Posting a patch which is a modification of the patch posted by [~ehiggs].
The test is now renamed to {{TestNamenodeStorageDirectives}} from
{{TestDataNodeStorage}} (also moved to the
{{org.apache.hadoop.hdfs.server.namenode}} package). Also, added a new
{{TestBlockPlacementPolicy}} and modified the existing
{{TestVolumeChoosingPolicy}} to ensure that the storage id passed to the
{{VolumeChoosingPolicy}} is exactly the same as chosen by the
{{BlockPlacementPolicy}}. Also fixed some of the checkstyle issues from the
earlier patch.
[~ehiggs] please take a look to see if this makes sense. I also removed the
{{verifyFileReplicasOnOnlyThreeStorageID}} as it was not being called, and
changed the visibility of some of the functions in the test to {{private}}.
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-9807:
-
Status: Patch Available (was: Open)
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch, HDFS-9807.008.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-9807:
-
Attachment: HDFS-9807.007.patch
Adding a patch that fixes check-style issues and adds a nose-tail test using
{{MiniDFSCluster}}.
Note: the test just checks that {storageId}} passed into the
{{VolumeChoosingPolicy}} is part of the volume list. It doesn't test that it's
the same as the one sent by the NN as part of the request. I wasn't sure of the
best way to get the value from the {{BlockPlacementPolicy}} into the
{{VolumeChoosingPolicy}} for comparison. If you think it's required and have a
good idea of how to connect them then let me know what you think.
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch, HDFS-9807.007.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-9807:
-
Attachment: HDFS-9807.006.patch
Attaching patch rebased now that HDFS-6708 has been merged.
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch,
> HDFS-9807.006.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-9807:
-
Attachment: HDFS-9807.005.patch
Attached a patch which fixes the javadoc issues.
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch, HDFS-9807.005.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-9807:
-
Attachment: HDFS-9807.004.patch
Attaching an updated patch. This fixes the findbugs issue as well as some check
style issues.
Some check style issues are left untouched since they are consistent with the
surrounding code or require considerable refactoring (parameter numbers, method
length).
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch, HDFS-9807.004.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-9807:
-
Attachment: HDFS-9807.003.patch
Attaching file that rebases on top of the updated HDFS-6708.006.patch and takes
[~virajith] and [~chris.douglas]'s comments into account.
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch,
> HDFS-9807.003.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HDFS-9807:
Attachment: HDFS-9807.002.patch
bq. I am planning to push the information like storageID and storageType
FsVolumeList (as described in the last suggestion in my previous comment) and
probably also into the interface for the VolumeChoosingPolicy, so it can decide
to choose the volume based on all the info.
+1 for this approach. The configured {{VolumeChoosingPolicy}} can elect to
ignore the particular storageid chosen by the NameNode, as it does now.
Uploaded a new patch with really minor changes (removing some vestigial code,
fixed findbugs, etc.). A couple questions:
* The {{BlockTokenIdentifier}} also updates the legacy, {{Writable}} token.
This wouldn't work with older clients?
* There's an assertion in {{StripedWriter}} that targetStorageIDs is not null;
does that always hold?
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch, HDFS-9807.002.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-9807:
-
Affects Version/s: 3.0.0-alpha2
Status: Patch Available (was: Open)
Attached a patch that threads the StorageID all the way through to the
{{FsVolumeList}} and then ignores it (since the NN isn't sending a useful
value).
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha2
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9807) Add an optional StorageID to writes
[
https://issues.apache.org/jira/browse/HDFS-9807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-9807:
-
Attachment: HDFS-9807.001.patch
Attaching a patch that threads the StorageID all the way through to the
{{FsVolumeList}} and then ignores it (since the NN isn't sending a useful
value).
> Add an optional StorageID to writes
> ---
>
> Key: HDFS-9807
> URL: https://issues.apache.org/jira/browse/HDFS-9807
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chris Douglas
>Assignee: Ewan Higgs
> Attachments: HDFS-9807.001.patch
>
>
> The {{BlockPlacementPolicy}} considers specific storages, but when the
> replica is written the DN {{VolumeChoosingPolicy}} is unaware of any
> preference or constraints from other policies affecting placement. This
> limits heterogeneity to the declared storage types, which are treated as
> fungible within the target DN. It should be possible to influence or
> constrain the DN policy to select a particular storage.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
