[
https://issues.apache.org/jira/browse/HDFS-14547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinglun updated HDFS-14547:
---------------------------
Description:
Our XiaoMi HDFS is considering upgrading from 2.6 to 3.1. We notice the storage
type quota 'tsCounts' is instantiated to
EnumCounters<StorageType>(StorageType.class), so it will cost a long[5] even if
we don't have any storage type quota on this inode(only space quota or name
quota).
In our cluster we have many dirs with quota and the NameNode's memory is in
tension, so the additional cost will be a problem.
See DirectoryWithQuotaFeature.Builder().
{code:java}
class DirectoryWithQuotaFeature$Builder {
public Builder() {
this.quota = new QuotaCounts.Builder().nameSpace(DEFAULT_NAMESPACE_QUOTA).
storageSpace(DEFAULT_STORAGE_SPACE_QUOTA).
typeSpaces(DEFAULT_STORAGE_SPACE_QUOTA).build();// set default value -1.
this.usage = new QuotaCounts.Builder().nameSpace(1).build();
}
public Builder typeSpaces(long val) {// set default value.
this.tsCounts.reset(val);
return this;
}
}
class QuotaCounts$Builder {
public Builder() {
this.nsSsCounts = new EnumCounters<Quota>(Quota.class);
this.tsCounts = new EnumCounters<StorageType>(StorageType.class);
}
}
class EnumCounters {
public EnumCounters(final Class<E> enumClass) {
final E[] enumConstants = enumClass.getEnumConstants();
Preconditions.checkNotNull(enumConstants);
this.enumClass = enumClass;
this.counters = new long[enumConstants.length];// new a long array here.
}
}
{code}
Related to HDFS-14542.
was:
Our XiaoMi HDFS is considering upgrading from 2.6 to 3.1. We notice the storage
type quota 'tsCounts' is instantiated to
EnumCounters<StorageType>(StorageType.class), so it will cost a long[5] even if
we don't have any storage type quota on this
inode(StorageType.class.getEnumConstants() == 5).
In our cluster we have many dirs with quota and the NameNode's memory is in
tension, so the additional cost will be a problem.
See DirectoryWithQuotaFeature.Builder().
{code:java}
class DirectoryWithQuotaFeature$Builder {
public Builder() {
this.quota = new QuotaCounts.Builder().nameSpace(DEFAULT_NAMESPACE_QUOTA).
storageSpace(DEFAULT_STORAGE_SPACE_QUOTA).
typeSpaces(DEFAULT_STORAGE_SPACE_QUOTA).build();// set default value -1.
this.usage = new QuotaCounts.Builder().nameSpace(1).build();
}
public Builder typeSpaces(long val) {// set default value.
this.tsCounts.reset(val);
return this;
}
}
class QuotaCounts$Builder {
public Builder() {
this.nsSsCounts = new EnumCounters<Quota>(Quota.class);
this.tsCounts = new EnumCounters<StorageType>(StorageType.class);
}
}
class EnumCounters {
public EnumCounters(final Class<E> enumClass) {
final E[] enumConstants = enumClass.getEnumConstants();
Preconditions.checkNotNull(enumConstants);
this.enumClass = enumClass;
this.counters = new long[enumConstants.length];// new a long array here.
}
}
{code}
Related to HDFS-14542.
> DirectoryWithQuotaFeature.quota costs additional memory even the storage type
> quota is not set.
> -----------------------------------------------------------------------------------------------
>
> Key: HDFS-14547
> URL: https://issues.apache.org/jira/browse/HDFS-14547
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.1.0
> Reporter: Jinglun
> Priority: Major
>
> Our XiaoMi HDFS is considering upgrading from 2.6 to 3.1. We notice the
> storage type quota 'tsCounts' is instantiated to
> EnumCounters<StorageType>(StorageType.class), so it will cost a long[5] even
> if we don't have any storage type quota on this inode(only space quota or
> name quota).
> In our cluster we have many dirs with quota and the NameNode's memory is in
> tension, so the additional cost will be a problem.
> See DirectoryWithQuotaFeature.Builder().
>
> {code:java}
> class DirectoryWithQuotaFeature$Builder {
> public Builder() {
> this.quota = new QuotaCounts.Builder().nameSpace(DEFAULT_NAMESPACE_QUOTA).
> storageSpace(DEFAULT_STORAGE_SPACE_QUOTA).
> typeSpaces(DEFAULT_STORAGE_SPACE_QUOTA).build();// set default value -1.
> this.usage = new QuotaCounts.Builder().nameSpace(1).build();
> }
> public Builder typeSpaces(long val) {// set default value.
> this.tsCounts.reset(val);
> return this;
> }
> }
> class QuotaCounts$Builder {
> public Builder() {
> this.nsSsCounts = new EnumCounters<Quota>(Quota.class);
> this.tsCounts = new EnumCounters<StorageType>(StorageType.class);
> }
> }
> class EnumCounters {
> public EnumCounters(final Class<E> enumClass) {
> final E[] enumConstants = enumClass.getEnumConstants();
> Preconditions.checkNotNull(enumConstants);
> this.enumClass = enumClass;
> this.counters = new long[enumConstants.length];// new a long array here.
> }
> }
> {code}
> Related to HDFS-14542.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
