[
https://issues.apache.org/jira/browse/HDFS-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yangzhanling updated HDFS-16416:
--------------------------------
Description:
Namenode was unavailable when high concurrency when meta data wrote to
DoubleBuffer,and the currentBuff was full (512k) ,the other cursyncBuffer was
happend sync to dist or journalnode,so we request to namenode was unavailable.
Because the currentBuffer size is 512k,it is smaller in some bigger request
cluster.but it is not updated by configuring with {*}hdfs-site.xml{*}.it is
hardcoding in Java source code and unreasonable.
as listed below:
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager#outputBufferCapacity
private int outputBufferCapacity = 512 * 1024;
so we can update sample:
//first add key and value in DFSConfigKeys
//org.apache.hadoop.hdfs.DFSConfigKeys
//hdfs-site.xml
public static final String DFS_NAMENODE_BUFFER_KEY = "dfs.namenode.buffer.key";
public static final int DFS_NAMENODE_BUFFER_VALUE_DEFAULT = 512 * 1024;
//second,add param from configurature
//org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager
this.outputBufferCapacity = conf.getInt(
DFSConfigKeys.DFS_NAMENODE_BUFFER_KEY,
DFSConfigKeys.DFS_NAMENODE_BUFFER_VALUE_DEFAULT
);
//third ,add config in hdfs-site.xml
<property>
<name>dfs.namenode.buffer.key"</name>
<value>1024</value>
</property>
was:
Namenode was unavailable when high concurrency when meta data wrote to
DoubleBuffer,and the currentBuff was full (512k) ,the other cursyncBuffer was
happend sync to dist or journalnode,so we request to namenode was unavailable.
Because the currentBuffer size is 512k,it is smaller in some bigger request
cluster.but it is not updated by configuring with *hdfs-site.xml*.it is
hardcoding in Java source code and unreasonable.
Environment: (was:
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager#outputBufferCapacity
private int outputBufferCapacity = 512 * 1024;
so we can update sample:
//first add key and value in DFSConfigKeys
//org.apache.hadoop.hdfs.DFSConfigKeys
//hdfs-site.xml
public static final String DFS_NAMENODE_BUFFER_KEY =
"dfs.namenode.buffer.key";
public static final int DFS_NAMENODE_BUFFER_VALUE_DEFAULT = 512 * 1024;
//second,add param from configurature
//org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager
this.outputBufferCapacity = conf.getInt(
DFSConfigKeys.DFS_NAMENODE_BUFFER_KEY,
DFSConfigKeys.DFS_NAMENODE_BUFFER_VALUE_DEFAULT
);
//third ,add config in hdfs-site.xml
<property>
<name>dfs.namenode.buffer.key"</name>
<value>1024</value>
</property>
)
> Namenode was unavailable when high concurrency
> ----------------------------------------------
>
> Key: HDFS-16416
> URL: https://issues.apache.org/jira/browse/HDFS-16416
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namanode
> Affects Versions: 2.7.7
> Reporter: yangzhanling
> Priority: Major
>
> Namenode was unavailable when high concurrency when meta data wrote to
> DoubleBuffer,and the currentBuff was full (512k) ,the other cursyncBuffer was
> happend sync to dist or journalnode,so we request to namenode was unavailable.
> Because the currentBuffer size is 512k,it is smaller in some bigger request
> cluster.but it is not updated by configuring with {*}hdfs-site.xml{*}.it is
> hardcoding in Java source code and unreasonable.
> as listed below:
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager#outputBufferCapacity
> private int outputBufferCapacity = 512 * 1024;
> so we can update sample:
> //first add key and value in DFSConfigKeys
> //org.apache.hadoop.hdfs.DFSConfigKeys
> //hdfs-site.xml
> public static final String DFS_NAMENODE_BUFFER_KEY =
> "dfs.namenode.buffer.key";
> public static final int DFS_NAMENODE_BUFFER_VALUE_DEFAULT = 512 * 1024;
> //second,add param from configurature
> //org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager
> this.outputBufferCapacity = conf.getInt(
> DFSConfigKeys.DFS_NAMENODE_BUFFER_KEY,
> DFSConfigKeys.DFS_NAMENODE_BUFFER_VALUE_DEFAULT
> );
> //third ,add config in hdfs-site.xml
> <property>
> <name>dfs.namenode.buffer.key"</name>
> <value>1024</value>
> </property>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
