[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: (was: HDFS-4988.03.patch) Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.05.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: (was: HDFS-4988.04.patch) Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.05.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.06.patch Updated patch to undo StorageID rename. Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.05.patch, HDFS-4988.06.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.06.patch Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.05.patch, HDFS-4988.06.patch, HDFS-4988.06.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: (was: HDFS-4988.06.patch) Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.05.patch, HDFS-4988.06.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.07.patch Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.05.patch, HDFS-4988.06.patch, HDFS-4988.07.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.07.patch Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.05.patch, HDFS-4988.06.patch, HDFS-4988.07.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.07.patch Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.05.patch, HDFS-4988.06.patch, HDFS-4988.07.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: (was: HDFS-4988.07.patch) Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.05.patch, HDFS-4988.06.patch, HDFS-4988.07.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: (was: HDFS-4988.07.patch) Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.05.patch, HDFS-4988.06.patch, HDFS-4988.07.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[
https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-4988:
Attachment: HDFS-4988.08.patch
{quote}
It seems that StorageDirectory.storageID should be moved to DataStorage.
{quote}
The DataStorage and StorageDirectory class organization is not easy to work
with. There is only one DataStorage object and that is used to manage all the
StorageDirectories. I don't see how to make StorageID a member of DataStorage
since it is different for each StorageDirectory. Do you mind if we leave it
this way for now, it would require refactoring of unrelated code to fix it.
Maybe we can do it in trunk later.
{quote}
In DataStorage, should cachedDatanodeUuid be moved to DataNode? Also, would
calling it as datanodeUuid good enough? For example, we use storageID (or
storageUuid) but not cachedStorageID.
{quote}
I think it needs to be a part of DataStorage. It is read from the VERSION file,
and needs an field like the remaining entries (I renamed it to DatanodeUuid)
{quote}
In new FsVolumeSpi.getStorageID() method, do you want to call it
getStorageUuid()?
{quote}
Yes I would like to rename it when we rename it everywhere else. I wanted it to
be consistent as much as possible for now. I filed HDFS-5264 for this.
{quote}
For the new perVolumeReplicaMap, it may be better to change ReplicaMap or add a
new class (say VolumeReplicaMap, which has a volume-to-ReplicaMap map) to
support the per-volume feature.
{quote}
Can I do this in a separate Jira? I agree with you it needs cleanup. I'll fix
it in HDFS-2832.
{quote}
The old FsDatasetSpi.getBlockReport(String bpid) method should be removed. It
is only used in tests.
{quote}
Do you mind if I remove it in a separate checkin when I fix the tests?
{quote}
I suggest not to manually fix the imports. We may use some IDE tools to fix
them (e.g. cmd-shit-o in eclipse.)
{quote}
This was done automatically by IntelliJ probably because new imports made the
list too long. I think it is a reasonable here since the list of imports would
be edited anyway.
Addressed all other remaining comments from both, thanks for the reviews
Nicholas and Junping.
Datanode must support all the volumes as individual storages
Key: HDFS-4988
URL: https://issues.apache.org/jira/browse/HDFS-4988
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Reporter: Suresh Srinivas
Assignee: Arpit Agarwal
Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch,
HDFS-4988.05.patch, HDFS-4988.06.patch, HDFS-4988.07.patch, HDFS-4988.08.patch
Currently all the volumes on datanode is reported as a single storage. This
change proposes reporting them as individual storage. This requires:
# A unique storage ID for each storage
#* This needs to be generated during formatting
# There should be an option to allow existing disks to be reported as single
storage unit for backward compatibility.
# A functionality is also needed to split the existing all volumes as single
storage unit to to individual storage units.
# -Configuration must allow for each storage unit a storage type attribute.
(Now HDFS-5000)-
# Block reports must be sent on a per storage basis. In some cases (such
memory tier) block reports may need to be sent more frequently. That means
block reporting period must be on a per storage type basis.
My proposal is for new clusters to configure volumes by default as separate
storage unit. Lets discuss.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4988: - Component/s: datanode Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.03.patch New patch depends on HDFS-5232. Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.03.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.04.patch Smaller patch by moving some changes to other Jiras. Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.03.patch, HDFS-4988.04.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.05.patch Some more patch cleanup to reduce the size of the change. Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch, HDFS-4988.03.patch, HDFS-4988.04.patch, HDFS-4988.05.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.02.patch Rebased patch. Test cases will need to be fixed. Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch, HDFS-4988.02.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Attachment: HDFS-4988.01.patch Preliminary patch. This needs some more work. This task requires more work than I anticipated. I am going to see if I can split it up into multiple patches. Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Arpit Agarwal Attachments: HDFS-4988.01.patch Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Description: Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. was: Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # Configuration must allow for each storage unit a storage type attribute. # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Arpit Agarwal Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # -Configuration must allow for each storage unit a storage type attribute. (Now HDFS-5000)- # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4988) Datanode must support all the volumes as individual storages
[ https://issues.apache.org/jira/browse/HDFS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4988: Assignee: Arpit Agarwal Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Arpit Agarwal Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # Configuration must allow for each storage unit a storage type attribute. # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
