[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Brandon Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13890139#comment-13890139
 ] 

Brandon Li commented on HDFS-5754:
--

[~vinayrpet], please feel free to use HDFS-5874 to fix the issue. Thanks!

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Fix For: HDFS-5535 (Rolling upgrades)
>
> Attachments: FeatureInfo.patch, HDFS-5754.001.patch, 
> HDFS-5754.002.patch, HDFS-5754.003.patch, HDFS-5754.004.patch, 
> HDFS-5754.006.patch, HDFS-5754.007.patch, HDFS-5754.008.patch, 
> HDFS-5754.009.patch, HDFS-5754.010.patch, HDFS-5754.012.patch, 
> HDFS-5754.013.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Brandon Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13889812#comment-13889812
 ] 

Brandon Li commented on HDFS-5754:
--

{quote}But LayoutVersion is split because both these could be different in same 
version of hadoop right?{quote}
Yes. Thanks for the catch, Vinay.
{quote}May be this log not required. Only below log is enough with condition 
removed.{quote}
Agree.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Fix For: HDFS-5535 (Rolling upgrades)
>
> Attachments: FeatureInfo.patch, HDFS-5754.001.patch, 
> HDFS-5754.002.patch, HDFS-5754.003.patch, HDFS-5754.004.patch, 
> HDFS-5754.006.patch, HDFS-5754.007.patch, HDFS-5754.008.patch, 
> HDFS-5754.009.patch, HDFS-5754.010.patch, HDFS-5754.012.patch, 
> HDFS-5754.013.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Vinay (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13887540#comment-13887540
 ] 

Vinay commented on HDFS-5754:
-

First of all, sorry for coming in late ( that too after commit )...

All Changes look good, except following one.

{noformat}+  if (DataNodeLayoutVersion.supports(
+  LayoutVersion.Feature.FEDERATION,
+  HdfsConstants.DATANODE_LAYOUT_VERSION) && 
+  HdfsConstants.DATANODE_LAYOUT_VERSION == nsInfo.getLayoutVersion()) {
 readProperties(sd, nsInfo.getLayoutVersion());
 writeProperties(sd);
 LOG.info("Layout version rolled back to " +{noformat}

Here while rolling back and previous directory not exist, 
{{DATANODE_LAYOUT_VERSION}} is directly compared with 
{{NAMENODE_LAYOUT_VERSION}} to update the VERSION file.
But LayoutVersion is split because both these could be different in same 
version of hadoop right?

{noformat}+if (HdfsConstants.DATANODE_LAYOUT_VERSION != 
nsInfo.getLayoutVersion()) {
+  LOG.info("DataNode and NameNode layout versions are different:" +
+" DataNode version: "+ HdfsConstants.DATANODE_LAYOUT_VERSION +
+" NameNode version: "+ nsInfo.getLayoutVersion());{noformat}
May be this log not required. Only below log is enough with condition removed.
{noformat}+if( HdfsConstants.DATANODE_LAYOUT_VERSION == 
nsInfo.getLayoutVersion())
+  LOG.info("Data-node version: " + HdfsConstants.DATANODE_LAYOUT_VERSION + 
+  " and name-node layout version: " + nsInfo.getLayoutVersion());{noformat}

If any changes required I am ok to do in separate Jira instead of re-opening 
this.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Fix For: HDFS-5535 (Rolling upgrades)
>
> Attachments: FeatureInfo.patch, HDFS-5754.001.patch, 
> HDFS-5754.002.patch, HDFS-5754.003.patch, HDFS-5754.004.patch, 
> HDFS-5754.006.patch, HDFS-5754.007.patch, HDFS-5754.008.patch, 
> HDFS-5754.009.patch, HDFS-5754.010.patch, HDFS-5754.012.patch, 
> HDFS-5754.013.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Brandon Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13887264#comment-13887264
 ] 

Brandon Li commented on HDFS-5754:
--

[~szetszwo], thanks for reviewing and updating the patch. The comments are 
addressed in the new patch.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: FeatureInfo.patch, HDFS-5754.001.patch, 
> HDFS-5754.002.patch, HDFS-5754.003.patch, HDFS-5754.004.patch, 
> HDFS-5754.006.patch, HDFS-5754.007.patch, HDFS-5754.008.patch, 
> HDFS-5754.009.patch, HDFS-5754.010.patch, HDFS-5754.012.patch, 
> HDFS-5754.013.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Tsz Wo (Nicholas), SZE (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13886680#comment-13886680
 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-5754:
--

Thanks a lot for update.  Patch looks good.  Some minor comments:

- When a StorageInfo object is created, the storage type is known and remains 
unchanged.  Thus, StorageInfo.storageType should be final.

- Let's add NodeType as a parameter to PBHelper.convert(StorageInfoProto info) 
since the caller knows the NodeType.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: FeatureInfo.patch, HDFS-5754.001.patch, 
> HDFS-5754.002.patch, HDFS-5754.003.patch, HDFS-5754.004.patch, 
> HDFS-5754.006.patch, HDFS-5754.007.patch, HDFS-5754.008.patch, 
> HDFS-5754.009.patch, HDFS-5754.010.patch, HDFS-5754.012.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Brandon Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13885944#comment-13885944
 ] 

Brandon Li commented on HDFS-5754:
--

{quote} In DataStorage, BPServiceActor and BlockPoolSliceStorage, it should not 
compare DATANODE_LAYOUT_VERSION with nsInfo.getLayoutVersion() anymore.{quote}
removed. 

{quote} Map> should be Map>. We should declear with interface Set (or should we use 
SortedSet?) instead of particular implementation TreeSet.{quote}
yes.
{quote} In PBHelper, could we use null (i.e. unknown) instead of 
NodeType.NAME_NODE as default? Or we could add a setStorageType(NodeType) 
method so that we could set it when it is null. {quote}
If we use null as default and add new method setStorageType() to set 
storageType in a few places after receiving StorageInfo from the wire, the code 
is not as clean as just sending StorageType in the RPC payload. But I will 
upload a patch with the "default null" first to show the change. 
 
{quote}The type parameter below is not used. Should it be removed?{quote}
yes.
 
 {quote} I suggest to move the layout version related code out from NameNode 
and DataNode to new classes, say NameNodeLayoutVersion and 
DataNodeLayoutVersion. {quote}
Agree. It's better to hide the maps in these two classes than exposing them 
everywhere.


> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: FeatureInfo.patch, HDFS-5754.001.patch, 
> HDFS-5754.002.patch, HDFS-5754.003.patch, HDFS-5754.004.patch, 
> HDFS-5754.006.patch, HDFS-5754.007.patch, HDFS-5754.008.patch, 
> HDFS-5754.009.patch, HDFS-5754.010.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Tsz Wo (Nicholas), SZE (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13885317#comment-13885317
 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-5754:
--

- In DataStorage, BPServiceActor and BlockPoolSliceStorage, it should not 
compare DATANODE_LAYOUT_VERSION with nsInfo.getLayoutVersion() anymore.

- Map> should be Map>.  We should declear with interface Set (or should we use 
SortedSet?) instead of particular implementation TreeSet.

- In PBHelper, could we use null (i.e. unknown) instead of NodeType.NAME_NODE 
as default?  Or we could add a setStorageType(NodeType) method so that we could 
set it when it is null.

- The type parameter below is not used.  Should it be removed?
{code}
//Storage.java
   protected Storage(NodeType type, StorageInfo storageInfo) {
 super(storageInfo);
-this.storageType = type;
   }
{code}

- I suggest to move the layout version related code out from NameNode and 
DataNode to new classes, say NameNodeLayoutVersion and DataNodeLayoutVersion.


> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: FeatureInfo.patch, HDFS-5754.001.patch, 
> HDFS-5754.002.patch, HDFS-5754.003.patch, HDFS-5754.004.patch, 
> HDFS-5754.006.patch, HDFS-5754.007.patch, HDFS-5754.008.patch, 
> HDFS-5754.009.patch, HDFS-5754.010.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Brandon Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13883115#comment-13883115
 ] 

Brandon Li commented on HDFS-5754:
--

{quote} I think we need two maps. Do you agree?{quote}
Yes. We need to maps here. 
Looks like it's hard to keep the patch at a minimal size. Uploaded a new patch.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: FeatureInfo.patch, HDFS-5754.001.patch, 
> HDFS-5754.002.patch, HDFS-5754.003.patch, HDFS-5754.004.patch, 
> HDFS-5754.006.patch, HDFS-5754.007.patch, HDFS-5754.008.patch, 
> HDFS-5754.009.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Tsz Wo (Nicholas), SZE (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880077#comment-13880077
 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-5754:
--

- The patch adds the new system properties (HDFS_SERVICE_LAYOUT etc.) for 
initializing LayoutVersion.map.  It seems that it won't work for unit tests 
since a test may run both NN and DN but we only have one map.  I think we need 
two maps.  Do you agree?

- If HdfsConstants.LAYOUT_VERSION is initialized with NameNode.Feature.values() 
as below, the code in DN using HdfsConstants.LAYOUT_VERSION is incorrect.  I 
guess it needs two LAYOUT_VERSIONs.
{code}
//HdfsConstants
   public static final int LAYOUT_VERSION = LayoutVersion
-  .getCurrentLayoutVersion();
+  .getCurrentLayoutVersion(NameNode.Feature.values());
{code}

- LayoutFeatureComparator.compare(..) could be simplified as below
{code}
public int compare(LayoutFeature arg0, LayoutFeature arg1) {
  return arg0.getLayoutVersion() - arg1.getLayoutVersion();
}
{code}

- In the patch, the Feature enum in LayoutVersion, NameNode and DataNode are 
very similar.  We could add a new class FeatureInfo for reducing repeated code 
as shown in FeatureInfo.patch.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: HDFS-5754.001.patch, HDFS-5754.002.patch, 
> HDFS-5754.003.patch, HDFS-5754.004.patch, HDFS-5754.006.patch, 
> HDFS-5754.007.patch, HDFS-5754.008.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Brandon Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879463#comment-13879463
 ] 

Brandon Li commented on HDFS-5754:
--

[~szetszwo], Thanks. I've uploaded a new patch which reverted the change you 
pointed out.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: HDFS-5754.001.patch, HDFS-5754.002.patch, 
> HDFS-5754.003.patch, HDFS-5754.004.patch, HDFS-5754.006.patch, 
> HDFS-5754.007.patch, HDFS-5754.008.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Tsz Wo (Nicholas), SZE (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879404#comment-13879404
 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-5754:
--

Thanks Brandon for updating the patch.  Some comments

- The patch changes some pre-rolling upgrade version check to use 
NameNode.layoutVersion.supports(..) instead of LayoutVersion.supports(..) such 
as below.  Some of the changes are in the common code of DN and NN, and some of 
the changes are in datanode code.  These changes are incorrect since DN should 
not use NameNode.layoutVersion.
-* e.g 1
{code}
//Storage.java
-if (LayoutVersion.supports(Feature.FEDERATION, layoutVersion)) {
+if (NameNode.layoutVersion.supports(Feature.FEDERATION, layoutVersion)) {
{code}
-* e.g. 2
{code}
//datanode/BlockPoolSliceStorage.java
-if (!LayoutVersion.supports(Feature.FEDERATION, layoutVersion)) {
+if (!NameNode.layoutVersion.supports(Feature.FEDERATION, layoutVersion)) {
{code}
I subject we keep the static LayoutVersion.supports(..) method and revert all 
the changes from LayoutVersion.supports(..) to 
NameNode.layoutVersion.supports(..). 

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: HDFS-5754.001.patch, HDFS-5754.002.patch, 
> HDFS-5754.003.patch, HDFS-5754.004.patch, HDFS-5754.006.patch, 
> HDFS-5754.007.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Tsz Wo (Nicholas), SZE (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878500#comment-13878500
 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-5754:
--

In the new patch, LayoutVersion.Feature is still moved to NameNode.  We should 
only move ROLLING_UPGRADE_MARKER to NameNode and keep the other features such 
as ADD_DATANODE_AND_STORAGE_UUIDS in LayoutVersion.  Obviously, 
ADD_DATANODE_AND_STORAGE_UUIDS is not a namenode feature.  Indeed, all the 
pre-rolling upgrade features (i.e. lv-1 - lv-51) are shared by NN and DN.  We 
cannot move them to NN.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: HDFS-5754.001.patch, HDFS-5754.002.patch, 
> HDFS-5754.003.patch, HDFS-5754.004.patch, HDFS-5754.006.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Brandon Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878056#comment-13878056
 ] 

Brandon Li commented on HDFS-5754:
--

My bad. Upload a new patch soon.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: HDFS-5754.001.patch, HDFS-5754.002.patch, 
> HDFS-5754.003.patch, HDFS-5754.004.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Tsz Wo (Nicholas), SZE (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877998#comment-13877998
 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-5754:
--

> It may be a good idea to add NameNode.Feature and move ROLLING_UPGRADE_MARKER 
> to there.

Sorry that I may not be clear: I mean that move only ROLLING_UPGRADE_MARKER to 
NameNode.Feature but leave the pre-rolling upgrade features in 
LayoutVersion.Feature since these features are shared by both NN and DN.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: HDFS-5754.001.patch, HDFS-5754.002.patch, 
> HDFS-5754.003.patch, HDFS-5754.004.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Tsz Wo (Nicholas), SZE (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13875523#comment-13875523
 ] 

Tsz Wo (Nicholas), SZE commented on HDFS-5754:
--

I like the idea of adding a new LayoutFeature interface and a new 
DataNodeFeature enum.  Some comments:

- We need to think more about upgrading.  DataNodeFeature should not start with 
-1 since -1 in already used previously.  ROLLING_UPGRADE_MARKER(-52) was added 
for rolling upgrade. DataNodeFeature should also start with -52.

- BTW, I think we should move DataNodeFeature to the datanode package or may be 
an inner class of DataNode (i.e. DataNode.Feature).

- It may be a good idea to add NameNode.Feature and move ROLLING_UPGRADE_MARKER 
to there.

- We should also add tests for different DataNode layout versions and different 
NameNode layout versions.  We may do it separately.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: HDFS-5754.001.patch, HDFS-5754.002.patch, 
> HDFS-5754.003.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Brandon Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873974#comment-13873974
 ] 

Brandon Li commented on HDFS-5754:
--

The unit test failure was not introduced by this patch. I've filed HDFS-5791 to 
track the unit test fix.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: HDFS-5754.001.patch, HDFS-5754.002.patch, 
> HDFS-5754.003.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873943#comment-13873943
 ] 

Hadoop QA commented on HDFS-5754:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12623435/HDFS-5754.003.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.hdfs.web.TestHttpsFileSystem

{color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-HDFS-Build/5900//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5900//console

This message is automatically generated.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: HDFS-5754.001.patch, HDFS-5754.002.patch, 
> HDFS-5754.003.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873103#comment-13873103
 ] 

Hadoop QA commented on HDFS-5754:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12623312/HDFS-5754.002.patch
  against trunk revision .

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 1 new 
or modified test files.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 eclipse:eclipse{color}.  The patch built with 
eclipse:eclipse.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 core tests{color}.  The patch failed these unit tests in 
hadoop-hdfs-project/hadoop-hdfs:

  org.apache.hadoop.hdfs.server.namenode.TestMetaSave
  
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength
  org.apache.hadoop.fs.TestFcHdfsCreateMkdir
  org.apache.hadoop.hdfs.TestSetTimes
  
org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks
  
org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
  org.apache.hadoop.hdfs.TestDFSStartupVersions
  org.apache.hadoop.fs.TestUrlStreamHandler
  
org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
  org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
  org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes
  org.apache.hadoop.hdfs.server.namenode.TestFsLimits
  
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshottableDirListing
  
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion
  org.apache.hadoop.fs.TestSymlinkHdfsDisable
  org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
  org.apache.hadoop.hdfs.server.namenode.TestNameNodeMXBean
  org.apache.hadoop.fs.TestGlobPaths
  
org.apache.hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
  
org.apache.hadoop.hdfs.server.blockmanagement.TestPendingReplication
  
org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
  org.apache.hadoop.hdfs.server.namenode.TestINodeFile
  
org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks
  org.apache.hadoop.hdfs.TestDFSShellGenericOptions
  org.apache.hadoop.hdfs.server.namenode.ha.TestHAMetrics
  org.apache.hadoop.hdfs.TestDatanodeDeath
  org.apache.hadoop.hdfs.qjournal.client.TestIPCLoggerChannel
  org.apache.hadoop.hdfs.TestFileAppend2
  
org.apache.hadoop.hdfs.server.namenode.TestCheckPointForSecurityTokens
  
org.apache.hadoop.hdfs.server.namenode.TestStartupOptionUpgrade
  org.apache.hadoop.hdfs.TestDFSPermission
  org.apache.hadoop.hdfs.TestFileAppendRestart
  org.apache.hadoop.hdfs.TestDataTransferProtocol
  org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
  org.apache.hadoop.hdfs.server.namenode.TestFileLimit
  org.apache.hadoop.hdfs.qjournal.server.TestJournalNode
  org.apache.hadoop.fs.TestSymlinkHdfsFileContext
  org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool
  org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
  org.apache.hadoop.fs.TestEnhancedByteBufferAccess
  org.apache.hadoop.hdfs.TestDFSUpgrade
  org.apache.hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA
  org.apache.hadoop.hdfs.TestIsMethodSupported
  org.apache.hadoop.fs.TestHDFSFileContextMainOperations
  org.apache.hadoop.hdfs.server.namenode.TestNameNodeRecovery
  org.apache.hadoop.hdfs.server.datanode.TestBlockReport
  org.apache.hadoop.hdfs.TestParallelUnixDomainRead
  org.apache.hadoop.hdfs.TestHDFSServerPorts
  
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotListing
  
org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManagerUnit
  
org.apache.hadoop

[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873041#comment-13873041
 ] 

Hadoop QA commented on HDFS-5754:
-

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12623307/HDFS-5754.001.patch
  against trunk revision .

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5893//console

This message is automatically generated.

> Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion 
> 
>
> Key: HDFS-5754
> URL: https://issues.apache.org/jira/browse/HDFS-5754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Brandon Li
> Attachments: HDFS-5754.001.patch
>
>
> Currently, LayoutVersion defines the on-disk data format and supported 
> features of the entire cluster including NN and DNs.  LayoutVersion is 
> persisted in both NN and DNs.  When a NN/DN starts up, it checks its 
> supported LayoutVersion against the on-disk LayoutVersion.  Also, a DN with a 
> different LayoutVersion than NN cannot register with the NN.
> We propose to split LayoutVersion into two independent values that are local 
> to the nodes:
> - NamenodeLayoutVersion - defines the on-disk data format in NN, including 
> the format of FSImage, editlog and the directory structure.
> - DatanodeLayoutVersion - defines the on-disk data format in DN, including 
> the format of block data file, metadata file, block pool layout, and the 
> directory structure.  
> The LayoutVersion check will be removed in DN registration.  If 
> NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling 
> upgrade, then only rollback is supported and downgrade is not.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)