[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-14247:
--
Fix Version/s: (was: 2.0.0)
2.0.0-alpha-4
3.0.0
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-4
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff, HBASE-14247.master.001.patch,
> HBASE-14247.master.002.patch, HBASE-14247.master.003.patch,
> HBASE-14247.master.004.patch, HBASE-14247.master.005.patch
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-14247:
---
Resolution: Fixed
Hadoop Flags: Incompatible change
Status: Resolved (was: Patch Available)
Pushed to master and branch-2. Thanks all for reviewing.
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff, HBASE-14247.master.001.patch,
> HBASE-14247.master.002.patch, HBASE-14247.master.003.patch,
> HBASE-14247.master.004.patch, HBASE-14247.master.005.patch
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-14247:
---
Attachment: HBASE-14247.master.005.patch
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff, HBASE-14247.master.001.patch,
> HBASE-14247.master.002.patch, HBASE-14247.master.003.patch,
> HBASE-14247.master.004.patch, HBASE-14247.master.005.patch
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-14247:
---
Release Note: Add a new config hbase.separate.oldlogdir.by.regionserver.
The default value is false. If this config is true, the old wal dir will be
separated by regionservers. This will change the oldWALs layout. The oldWALs is
used by replication. So if a cluster didn't use replication, it can be rolling
upgrade (upgrade this config from false to true) directly. If a cluster use
replication, the oldWALs will be not found when layout changed. So the cluster
need rolling upgrade twice. Firstly, only rolling cluster to use new version
code. Secondly rolling the config from false to true. Because the cluster
already rolling to new version code, so it can find the oldWALs in the new dir
layout.
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247.master.001.patch,
> HBASE-14247.master.002.patch, HBASE-14247.master.003.patch,
> HBASE-14247.master.004.patch, HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-14247:
---
Attachment: HBASE-14247.master.004.patch
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247.master.001.patch,
> HBASE-14247.master.002.patch, HBASE-14247.master.003.patch,
> HBASE-14247.master.004.patch, HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-14247:
---
Attachment: HBASE-14247.master.003.patch
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247.master.001.patch,
> HBASE-14247.master.002.patch, HBASE-14247.master.003.patch,
> HBASE-14247-v001.diff, HBASE-14247-v002.diff, HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-14247:
---
Attachment: HBASE-14247.master.002.patch
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247.master.001.patch,
> HBASE-14247.master.002.patch, HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-14247:
---
Status: Patch Available (was: In Progress)
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247.master.001.patch, HBASE-14247-v001.diff,
> HBASE-14247-v002.diff, HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-14247:
---
Attachment: HBASE-14247.master.001.patch
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247.master.001.patch, HBASE-14247-v001.diff,
> HBASE-14247-v002.diff, HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-14247:
---
Status: In Progress (was: Patch Available)
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-14247:
--
Priority: Critical (was: Minor)
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-14247:
Attachment: HBASE-14247-v003.diff
Fix the checkstyle problems
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff,
> HBASE-14247-v003.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-14247:
Component/s: wal
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-14247:
Attachment: HBASE-14247-v002.diff
Fix some failed tests
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff, HBASE-14247-v002.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-14247:
Fix Version/s: 2.0.0
Status: Patch Available (was: Open)
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-14247:
Attachment: HBASE-14247-v001.diff
First patch to trigger large tests
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14247-v001.diff
>
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-14247:
Description:
Currently all old WALs of regionservers are achieved into the single directory
of oldWALs. In big clusters, because of long TTL of WAL or disabled
replications, the number of files under oldWALs may reach the
max-directory-items limit of HDFS, which will make the hbase cluster crashed.
{quote}
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
limit=1048576 items=1048576
{quote}
A simple solution is to separate the old WALs into different directories
according to the server name of the WAL.
Suggestions are welcomed~ Thanks
was:
Currently all old WALs of regionservers are achieved into the single directory
of oldWALs. In big clusters, because of long TTL of WAL or disabled
replications, the number of files under oldWALs may reach the
max-directory-items limit of HDFS, which will make the hbase cluster crashed.
```
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
limit=1048576 items=1048576
```
A simple solution is to separate the old WALs into different directories
according to the server name of the WAL.
Suggestions are welcomed~ Thanks
> Separate the old WALs into different regionserver directories
> -
>
> Key: HBASE-14247
> URL: https://issues.apache.org/jira/browse/HBASE-14247
> Project: HBase
> Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
>
> Currently all old WALs of regionservers are achieved into the single
> directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
> replications, the number of files under oldWALs may reach the
> max-directory-items limit of HDFS, which will make the hbase cluster crashed.
> {quote}
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
> The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
> limit=1048576 items=1048576
> {quote}
> A simple solution is to separate the old WALs into different directories
> according to the server name of the WAL.
> Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
