[jira] [Commented] (HDFS-5484) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847588#comment-13847588
]
Eric Sirianni commented on HDFS-5484:
-
This fix was basically nullified by the following change made via HDFS-5542
{code}
+ DatanodeStorageInfo updateStorage(DatanodeStorage s) {
synchronized (storageMap) {
DatanodeStorageInfo storage = storageMap.get(s.getStorageID());
if (storage == null) {
@@ -670,8 +658,6 @@
for DN + getXferAddr());
storage = new DatanodeStorageInfo(this, s);
storageMap.put(s.getStorageID(), storage);
- } else {
-storage.setState(s.getState());
}
return storage;
}
{code}
Is there a reason that 'else' was removed? By no longer updating the state in
the {{BlockReport}} processing path, we effectively get the bogus state type
that is set via the first heartbeat (see the fix for HDFS-5455):
{code}
+ if (storage == null) {
+// This is seen during cluster initialization when the heartbeat
+// is received before the initial block reports from each storage.
+storage = updateStorage(new DatanodeStorage(report.getStorageID()));
{code}
Even reverting the change and reintroducing the 'else' leaves the state type
temporarily inaccurate until the first block report. Wouldn't a better fix be
to simply include the full {{DatanodeStorage}} object in the {{StorageReport}}
(as opposed to only the Storage ID)?
As a matter of bookkeeping, should I reopen this JIRA, or would you prefer a
new one be created?
StorageType and State in DatanodeStorageInfo in NameNode is not accurate
Key: HDFS-5484
URL: https://issues.apache.org/jira/browse/HDFS-5484
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Eric Sirianni
Fix For: Heterogeneous Storage (HDFS-2832)
Attachments: HDFS-5484-HDFS-2832--2.patch, HDFS-5484-HDFS-2832.patch
The fields in DatanodeStorageInfo are updated from two distinct paths:
# block reports
# storage reports (via heartbeats)
The {{state}} and {{storageType}} fields are updated via the Block Report.
However, as seen in the code blow, these fields are populated from a dummy
{{DatanodeStorage}} object constructed in the DataNode:
{code}
BPServiceActor.blockReport() {
//...
// Dummy DatanodeStorage object just for sending the block report.
DatanodeStorage dnStorage = new DatanodeStorage(storageID);
//...
}
{code}
The net effect is that the {{state}} and {{storageType}} fields are always
the default of {{NORMAL}} and {{DISK}} in the NameNode.
The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
{code}
public MapString, BlockListAsLongs getBlockReports(String bpid);
{code}
to:
{code}
public MapDatanodeStorage, BlockListAsLongs getBlockReports(String bpid);
{code}
thereby allowing {{BPServiceActor}} to send the real {{DatanodeStorage}}
object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5484) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847823#comment-13847823
]
Arpit Agarwal commented on HDFS-5484:
-
Let's create a new Jira.
I am fine with adding the {{DatanodeStorage}} object to {{StorageReportProto}}.
It needs to be a new optional field and we cannot remove the existing
{{StorageUuid}} for protocol compatibility.
StorageType and State in DatanodeStorageInfo in NameNode is not accurate
Key: HDFS-5484
URL: https://issues.apache.org/jira/browse/HDFS-5484
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Eric Sirianni
Fix For: Heterogeneous Storage (HDFS-2832)
Attachments: HDFS-5484-HDFS-2832--2.patch, HDFS-5484-HDFS-2832.patch
The fields in DatanodeStorageInfo are updated from two distinct paths:
# block reports
# storage reports (via heartbeats)
The {{state}} and {{storageType}} fields are updated via the Block Report.
However, as seen in the code blow, these fields are populated from a dummy
{{DatanodeStorage}} object constructed in the DataNode:
{code}
BPServiceActor.blockReport() {
//...
// Dummy DatanodeStorage object just for sending the block report.
DatanodeStorage dnStorage = new DatanodeStorage(storageID);
//...
}
{code}
The net effect is that the {{state}} and {{storageType}} fields are always
the default of {{NORMAL}} and {{DISK}} in the NameNode.
The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
{code}
public MapString, BlockListAsLongs getBlockReports(String bpid);
{code}
to:
{code}
public MapDatanodeStorage, BlockListAsLongs getBlockReports(String bpid);
{code}
thereby allowing {{BPServiceActor}} to send the real {{DatanodeStorage}}
object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5484) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13827673#comment-13827673
]
Hadoop QA commented on HDFS-5484:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614889/HDFS-5484-HDFS-2832.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5503//console
This message is automatically generated.
StorageType and State in DatanodeStorageInfo in NameNode is not accurate
Key: HDFS-5484
URL: https://issues.apache.org/jira/browse/HDFS-5484
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Eric Sirianni
Attachments: HDFS-5484-HDFS-2832.patch
The fields in DatanodeStorageInfo are updated from two distinct paths:
# block reports
# storage reports (via heartbeats)
The {{state}} and {{storageType}} fields are updated via the Block Report.
However, as seen in the code blow, these fields are populated from a dummy
{{DatanodeStorage}} object constructed in the DataNode:
{code}
BPServiceActor.blockReport() {
//...
// Dummy DatanodeStorage object just for sending the block report.
DatanodeStorage dnStorage = new DatanodeStorage(storageID);
//...
}
{code}
The net effect is that the {{state}} and {{storageType}} fields are always
the default of {{NORMAL}} and {{DISK}} in the NameNode.
The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
{code}
public MapString, BlockListAsLongs getBlockReports(String bpid);
{code}
to:
{code}
public MapDatanodeStorage, BlockListAsLongs getBlockReports(String bpid);
{code}
thereby allowing {{BPServiceActor}} to send the real {{DatanodeStorage}}
object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5484) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13827678#comment-13827678
]
Eric Sirianni commented on HDFS-5484:
-
It appears that Jenkins tried to apply the patch against {{trunk}}. I used the
naming convention outlined
[here|http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch] to
identify the target branch for the patch, but it didn't seem to work.
StorageType and State in DatanodeStorageInfo in NameNode is not accurate
Key: HDFS-5484
URL: https://issues.apache.org/jira/browse/HDFS-5484
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Eric Sirianni
Attachments: HDFS-5484-HDFS-2832.patch
The fields in DatanodeStorageInfo are updated from two distinct paths:
# block reports
# storage reports (via heartbeats)
The {{state}} and {{storageType}} fields are updated via the Block Report.
However, as seen in the code blow, these fields are populated from a dummy
{{DatanodeStorage}} object constructed in the DataNode:
{code}
BPServiceActor.blockReport() {
//...
// Dummy DatanodeStorage object just for sending the block report.
DatanodeStorage dnStorage = new DatanodeStorage(storageID);
//...
}
{code}
The net effect is that the {{state}} and {{storageType}} fields are always
the default of {{NORMAL}} and {{DISK}} in the NameNode.
The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
{code}
public MapString, BlockListAsLongs getBlockReports(String bpid);
{code}
to:
{code}
public MapDatanodeStorage, BlockListAsLongs getBlockReports(String bpid);
{code}
thereby allowing {{BPServiceActor}} to send the real {{DatanodeStorage}}
object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5484) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13827881#comment-13827881
]
Arpit Agarwal commented on HDFS-5484:
-
Thanks for the contribution Eric. I don't think Jenkins can test patches out of
trunk.
Minor issues:
# Extra indentation in body of {{FsVolumeImpl#toDatanodeStorage}}.
{code}
+ DatanodeStorage toDatanodeStorage() {
+ return new DatanodeStorage(storageID, DatanodeStorage.State.NORMAL,
storageType);
+ }
{code}
# Javadoc for {{FsDatasetSpi#getBlockReports}} needs update for return value.
Since this is new functionality you can consider adding a test case to make
sure the storage type is propagated from DN to NN and then to client via
LocatedBlock.
StorageType and State in DatanodeStorageInfo in NameNode is not accurate
Key: HDFS-5484
URL: https://issues.apache.org/jira/browse/HDFS-5484
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Eric Sirianni
Attachments: HDFS-5484-HDFS-2832.patch
The fields in DatanodeStorageInfo are updated from two distinct paths:
# block reports
# storage reports (via heartbeats)
The {{state}} and {{storageType}} fields are updated via the Block Report.
However, as seen in the code blow, these fields are populated from a dummy
{{DatanodeStorage}} object constructed in the DataNode:
{code}
BPServiceActor.blockReport() {
//...
// Dummy DatanodeStorage object just for sending the block report.
DatanodeStorage dnStorage = new DatanodeStorage(storageID);
//...
}
{code}
The net effect is that the {{state}} and {{storageType}} fields are always
the default of {{NORMAL}} and {{DISK}} in the NameNode.
The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
{code}
public MapString, BlockListAsLongs getBlockReports(String bpid);
{code}
to:
{code}
public MapDatanodeStorage, BlockListAsLongs getBlockReports(String bpid);
{code}
thereby allowing {{BPServiceActor}} to send the real {{DatanodeStorage}}
object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5484) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13827928#comment-13827928
]
Eric Sirianni commented on HDFS-5484:
-
Sure. I will make those changes and re-upload the patch.
StorageType and State in DatanodeStorageInfo in NameNode is not accurate
Key: HDFS-5484
URL: https://issues.apache.org/jira/browse/HDFS-5484
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Eric Sirianni
Attachments: HDFS-5484-HDFS-2832.patch
The fields in DatanodeStorageInfo are updated from two distinct paths:
# block reports
# storage reports (via heartbeats)
The {{state}} and {{storageType}} fields are updated via the Block Report.
However, as seen in the code blow, these fields are populated from a dummy
{{DatanodeStorage}} object constructed in the DataNode:
{code}
BPServiceActor.blockReport() {
//...
// Dummy DatanodeStorage object just for sending the block report.
DatanodeStorage dnStorage = new DatanodeStorage(storageID);
//...
}
{code}
The net effect is that the {{state}} and {{storageType}} fields are always
the default of {{NORMAL}} and {{DISK}} in the NameNode.
The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
{code}
public MapString, BlockListAsLongs getBlockReports(String bpid);
{code}
to:
{code}
public MapDatanodeStorage, BlockListAsLongs getBlockReports(String bpid);
{code}
thereby allowing {{BPServiceActor}} to send the real {{DatanodeStorage}}
object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5484) StorageType and State in DatanodeStorageInfo in NameNode is not accurate
[
https://issues.apache.org/jira/browse/HDFS-5484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13817584#comment-13817584
]
Arpit Agarwal commented on HDFS-5484:
-
I made it a sub-task of HDFS-2832. Thanks Eric.
StorageType and State in DatanodeStorageInfo in NameNode is not accurate
Key: HDFS-5484
URL: https://issues.apache.org/jira/browse/HDFS-5484
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: Heterogeneous Storage (HDFS-2832)
Reporter: Eric Sirianni
The fields in DatanodeStorageInfo are updated from two distinct paths:
# block reports
# storage reports (via heartbeats)
The {{state}} and {{storageType}} fields are updated via the Block Report.
However, as seen in the code blow, these fields are populated from a dummy
{{DatanodeStorage}} object constructed in the DataNode:
{code}
BPServiceActor.blockReport() {
//...
// Dummy DatanodeStorage object just for sending the block report.
DatanodeStorage dnStorage = new DatanodeStorage(storageID);
//...
}
{code}
The net effect is that the {{state}} and {{storageType}} fields are always
the default of {{NORMAL}} and {{DISK}} in the NameNode.
The recommended fix is to change {{FsDatasetSpi.getBlockReports()}} from:
{code}
public MapString, BlockListAsLongs getBlockReports(String bpid);
{code}
to:
{code}
public MapDatanodeStorage, BlockListAsLongs getBlockReports(String bpid);
{code}
thereby allowing {{BPServiceActor}} to send the real {{DatanodeStorage}}
object with the block report.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
