[jira] [Updated] (HDFS-9282) Make data directory count and storage raw capacity related tests FsDataset-agnostic
[ https://issues.apache.org/jira/browse/HDFS-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-9282: -- Component/s: (was: HDFS) > Make data directory count and storage raw capacity related tests > FsDataset-agnostic > --- > > Key: HDFS-9282 > URL: https://issues.apache.org/jira/browse/HDFS-9282 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 2.7.1 >Reporter: Tony Wu >Assignee: Tony Wu >Priority: Minor > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9282.001.patch, HDFS-9282.002.patch, > HDFS-9282.003.patch, HDFS-9282.004.patch > > > DFSMiniCluster and several tests have hard coded assumption of the underlying > storage having 2 data directories (volumes). As HDFS-9188 pointed out, with > new FsDataset implementations, these hard coded assumption about number of > data directories and raw capacities of storage may change as well. > We need to extend FsDatasetTestUtils to provide: > * Number of data directories of underlying storage per DataNode > * Raw storage capacity of underlying storage per DataNode. > * Have MiniDFSCluster automatically pick up the correct values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9282) Make data directory count and storage raw capacity related tests FsDataset-agnostic
[ https://issues.apache.org/jira/browse/HDFS-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-9282: Resolution: Fixed Fix Version/s: 2.8.0 3.0.0 Status: Resolved (was: Patch Available) > Make data directory count and storage raw capacity related tests > FsDataset-agnostic > --- > > Key: HDFS-9282 > URL: https://issues.apache.org/jira/browse/HDFS-9282 > Project: Hadoop HDFS > Issue Type: Improvement > Components: HDFS, test >Affects Versions: 2.7.1 >Reporter: Tony Wu >Assignee: Tony Wu >Priority: Minor > Fix For: 3.0.0, 2.8.0 > > Attachments: HDFS-9282.001.patch, HDFS-9282.002.patch, > HDFS-9282.003.patch, HDFS-9282.004.patch > > > DFSMiniCluster and several tests have hard coded assumption of the underlying > storage having 2 data directories (volumes). As HDFS-9188 pointed out, with > new FsDataset implementations, these hard coded assumption about number of > data directories and raw capacities of storage may change as well. > We need to extend FsDatasetTestUtils to provide: > * Number of data directories of underlying storage per DataNode > * Raw storage capacity of underlying storage per DataNode. > * Have MiniDFSCluster automatically pick up the correct values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9282) Make data directory count and storage raw capacity related tests FsDataset-agnostic
[
https://issues.apache.org/jira/browse/HDFS-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tony Wu updated HDFS-9282:
--
Attachment: HDFS-9282.004.patch
In v4 patch:
* Addressed [~eddyxu]'s review comments (use {{try-with-resources}} to avoid
leaking resources).
> Make data directory count and storage raw capacity related tests
> FsDataset-agnostic
> ---
>
> Key: HDFS-9282
> URL: https://issues.apache.org/jira/browse/HDFS-9282
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: HDFS, test
>Affects Versions: 2.7.1
>Reporter: Tony Wu
>Assignee: Tony Wu
>Priority: Minor
> Attachments: HDFS-9282.001.patch, HDFS-9282.002.patch,
> HDFS-9282.003.patch, HDFS-9282.004.patch
>
>
> DFSMiniCluster and several tests have hard coded assumption of the underlying
> storage having 2 data directories (volumes). As HDFS-9188 pointed out, with
> new FsDataset implementations, these hard coded assumption about number of
> data directories and raw capacities of storage may change as well.
> We need to extend FsDatasetTestUtils to provide:
> * Number of data directories of underlying storage per DataNode
> * Raw storage capacity of underlying storage per DataNode.
> * Have MiniDFSCluster automatically pick up the correct values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9282) Make data directory count and storage raw capacity related tests FsDataset-agnostic
[ https://issues.apache.org/jira/browse/HDFS-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Wu updated HDFS-9282: -- Attachment: HDFS-9282.003.patch In v3 patch: * Rebased patch on latest trunk. > Make data directory count and storage raw capacity related tests > FsDataset-agnostic > --- > > Key: HDFS-9282 > URL: https://issues.apache.org/jira/browse/HDFS-9282 > Project: Hadoop HDFS > Issue Type: Improvement > Components: HDFS, test >Affects Versions: 2.7.1 >Reporter: Tony Wu >Assignee: Tony Wu >Priority: Minor > Attachments: HDFS-9282.001.patch, HDFS-9282.002.patch, > HDFS-9282.003.patch > > > DFSMiniCluster and several tests have hard coded assumption of the underlying > storage having 2 data directories (volumes). As HDFS-9188 pointed out, with > new FsDataset implementations, these hard coded assumption about number of > data directories and raw capacities of storage may change as well. > We need to extend FsDatasetTestUtils to provide: > * Number of data directories of underlying storage per DataNode > * Raw storage capacity of underlying storage per DataNode. > * Have MiniDFSCluster automatically pick up the correct values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9282) Make data directory count and storage raw capacity related tests FsDataset-agnostic
[ https://issues.apache.org/jira/browse/HDFS-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Wu updated HDFS-9282: -- Attachment: HDFS-9282.002.patch Addressed [~eddyxu]'s review comments. > Make data directory count and storage raw capacity related tests > FsDataset-agnostic > --- > > Key: HDFS-9282 > URL: https://issues.apache.org/jira/browse/HDFS-9282 > Project: Hadoop HDFS > Issue Type: Improvement > Components: HDFS, test >Affects Versions: 2.7.1 >Reporter: Tony Wu >Assignee: Tony Wu >Priority: Minor > Attachments: HDFS-9282.001.patch, HDFS-9282.002.patch > > > DFSMiniCluster and several tests have hard coded assumption of the underlying > storage having 2 data directories (volumes). As HDFS-9188 pointed out, with > new FsDataset implementations, these hard coded assumption about number of > data directories and raw capacities of storage may change as well. > We need to extend FsDatasetTestUtils to provide: > * Number of data directories of underlying storage per DataNode > * Raw storage capacity of underlying storage per DataNode. > * Have MiniDFSCluster automatically pick up the correct values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9282) Make data directory count and storage raw capacity related tests FsDataset-agnostic
[ https://issues.apache.org/jira/browse/HDFS-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Wu updated HDFS-9282: -- Status: Patch Available (was: Open) > Make data directory count and storage raw capacity related tests > FsDataset-agnostic > --- > > Key: HDFS-9282 > URL: https://issues.apache.org/jira/browse/HDFS-9282 > Project: Hadoop HDFS > Issue Type: Improvement > Components: HDFS, test >Affects Versions: 2.7.1 >Reporter: Tony Wu >Assignee: Tony Wu >Priority: Minor > Attachments: HDFS-9282.001.patch > > > DFSMiniCluster and several tests have hard coded assumption of the underlying > storage having 2 data directories (volumes). As HDFS-9188 pointed out, with > new FsDataset implementations, these hard coded assumption about number of > data directories and raw capacities of storage may change as well. > We need to extend FsDatasetTestUtils to provide: > * Number of data directories of underlying storage per DataNode > * Raw storage capacity of underlying storage per DataNode. > * Have MiniDFSCluster automatically pick up the correct values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9282) Make data directory count and storage raw capacity related tests FsDataset-agnostic
[ https://issues.apache.org/jira/browse/HDFS-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Wu updated HDFS-9282: -- Attachment: HDFS-9282.001.patch In this patch: * Add getNumOfDataDirs() and getRawCapacity() interfaces. * Have MiniDFSCluster automatically pick up the correct number of data directories. * Updated a few tests where we have hard coded values. > Make data directory count and storage raw capacity related tests > FsDataset-agnostic > --- > > Key: HDFS-9282 > URL: https://issues.apache.org/jira/browse/HDFS-9282 > Project: Hadoop HDFS > Issue Type: Improvement > Components: HDFS, test >Affects Versions: 2.7.1 >Reporter: Tony Wu >Assignee: Tony Wu >Priority: Minor > Attachments: HDFS-9282.001.patch > > > DFSMiniCluster and several tests have hard coded assumption of the underlying > storage having 2 data directories (volumes). As HDFS-9188 pointed out, with > new FsDataset implementations, these hard coded assumption about number of > data directories and raw capacities of storage may change as well. > We need to extend FsDatasetTestUtils to provide: > * Number of data directories of underlying storage per DataNode > * Raw storage capacity of underlying storage per DataNode. > * Have MiniDFSCluster automatically pick up the correct values. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
