[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Target Version/s: 2.4.0
Fix Version/s: 2.4.0
Merged to branch-2. Target version set to 2.4.0.
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Assignee: Arpit Agarwal
Priority: Minor
Fix For: 3.0.0, 2.4.0
Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
HDFS-5454.03.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Resolution: Fixed
Fix Version/s: 3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Junping (and congrats on your committer +1). I committed it to trunk.
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Assignee: Arpit Agarwal
Priority: Minor
Fix For: 3.0.0
Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
HDFS-5454.03.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HDFS-5454:
-
Assignee: Arpit Agarwal
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Assignee: Arpit Agarwal
Priority: Minor
Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
HDFS-5454.03.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Attachment: HDFS-5454.03.patch
Update patch to add test case.
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Priority: Minor
Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch,
HDFS-5454.03.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Attachment: HDFS-5454.02.patch
Updated patch.
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Priority: Minor
Attachments: HDFS-5454.01.patch, HDFS-5454.02.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Attachment: HDFS-5454.01.patch
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Priority: Minor
Attachments: HDFS-5454.01.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Affects Version/s: (was: 2.2.0)
3.0.0
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Priority: Minor
Attachments: HDFS-5454.01.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Issue Type: Sub-task (was: Bug)
Parent: HDFS-2832
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Priority: Minor
Attachments: HDFS-5454.01.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Status: Patch Available (was: Open)
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 3.0.0
Reporter: Eric Sirianni
Priority: Minor
Attachments: HDFS-5454.01.patch
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Affects Version/s: (was: 2.2.0)
Heterogeneous Storage (HDFS-2832)
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Eric Sirianni
Priority: Minor
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Assignee: (was: Arpit Agarwal)
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Eric Sirianni
Priority: Minor
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HDFS-5454) DataNode UUID should be assigned prior to FsDataset initialization
[
https://issues.apache.org/jira/browse/HDFS-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-5454:
Affects Version/s: (was: Heterogeneous Storage (HDFS-2832))
2.2.0
DataNode UUID should be assigned prior to FsDataset initialization
--
Key: HDFS-5454
URL: https://issues.apache.org/jira/browse/HDFS-5454
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.2.0
Reporter: Eric Sirianni
Priority: Minor
The DataNode's UUID ({{DataStorage.getDatanodeUuid()}} field) is NULL at the
point where the {{FsDataset}} object is created ({{DataNode.initStorage()}}.
As the {{DataStorage}} object is an input to the {{FsDataset}} factory
method, it is desirable for it to be fully populated with a UUID at this
point. In particular, our {{FsDatasetSpi}} implementation relies upon the
DataNode UUID as a key to access our underlying block storage device.
This also appears to be a regression compared to Hadoop 1.x - our 1.x
{{FSDatasetInterface}} plugin has a non-NULL UUID on startup. I haven't
fully traced through the code, but I suspect this came from the
{{BPOfferService}}/{{BPServiceActor}} refactoring to support federated
namenodes.
With HDFS-5448, the DataNode is now responsible for generating its own UUID.
This greatly simplifies the fix. Move the UUID check/generation in from
{{DataNode.createBPRegistration()}} to {{DataNode.initStorage()}}. This more
naturally co-locates UUID generation immediately subsequent to the read of
the UUID from the {{DataStorage}} properties file.
{code}
private void initStorage(final NamespaceInfo nsInfo) throws IOException {
// ...
final String bpid = nsInfo.getBlockPoolID();
//read storage info, lock data dirs and transition fs state if necessary
storage.recoverTransitionRead(this, bpid, nsInfo, dataDirs, startOpt);
// SUGGESTED NEW PLACE TO CHECK DATANODE UUID
checkDatanodeUuid();
// ...
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
