[
https://issues.apache.org/jira/browse/HDFS-16191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17412107#comment-17412107
]
Xing Lin commented on HDFS-16191:
---------------------------------
Hi [~prasad-acit],
Thanks for working on this!
As you asked in the github pull request, we don't support partitions larger
than NUM_RANGES_STATIC right now.
The key of a inode is calculated and then modulo by NUM_RANGES_STATIC in
indexof(). As a result, any partition that has an id larger than
NUM_RANGES_STATIC will receive no insertion.
If we want to support dynamic partition numbers, we need to modify indexof()
implementation as well. We need to replace `& (INodeMap.NUM_RANGES_STATIC -1)`
with something like `% partition_num`. Also note, indexof() is a static
function which means we can not access instance variable from here. I don't
know how to handle it now.
{code:java}
public static long indexOf(long[] key) {
if(key[key.length-1] == INodeId.ROOT_INODE_ID) {
return key[0];
}
long idx = LARGE_PRIME * key[0];
idx = (idx ^ (idx >> 32)) & (INodeMap.NUM_RANGES_STATIC -1);
return idx;
}
{code}
> [FGL] Fix FSImage loading issues on dynamic partitions
> ------------------------------------------------------
>
> Key: HDFS-16191
> URL: https://issues.apache.org/jira/browse/HDFS-16191
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Renukaprasad C
> Assignee: Renukaprasad C
> Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When new partitions gets added into PartitionGSet, iterator do not consider
> the new partitions. Which always iterate on Static Partition count. This lead
> to full of warn messages as below.
> 2021-08-28 03:23:19,420 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139780 when saving the leases.
> 2021-08-28 03:23:19,420 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139781 when saving the leases.
> 2021-08-28 03:23:19,420 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139784 when saving the leases.
> 2021-08-28 03:23:19,420 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139785 when saving the leases.
> 2021-08-28 03:23:19,420 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139786 when saving the leases.
> 2021-08-28 03:23:19,420 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139788 when saving the leases.
> 2021-08-28 03:23:19,421 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139789 when saving the leases.
> 2021-08-28 03:23:19,421 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139790 when saving the leases.
> 2021-08-28 03:23:19,421 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139791 when saving the leases.
> 2021-08-28 03:23:19,421 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139793 when saving the leases.
> 2021-08-28 03:23:19,421 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139795 when saving the leases.
> 2021-08-28 03:23:19,422 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139796 when saving the leases.
> 2021-08-28 03:23:19,422 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139797 when saving the leases.
> 2021-08-28 03:23:19,422 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139800 when saving the leases.
> 2021-08-28 03:23:19,422 WARN namenode.FSImageFormatPBINode: Fail to find
> inode 139801 when saving the leases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
