[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14008467#comment-14008467 ] Hudson commented on HBASE-11226: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #295 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/295/]) HBASE-11226 Document and increase the default value for hbase.hstore.flusher.count (nkeywal: rev 996705c61de1c7b533024dcb41cc571531ebfbb9) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java * hbase-common/src/main/resources/hbase-default.xml > Document and increase the default value for hbase.hstore.flusher.count > -- > > Key: HBASE-11226 > URL: https://issues.apache.org/jira/browse/HBASE-11226 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.99.0, 0.98.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.99.0, 0.98.3 > > Attachments: 11226.v1.patch > > > HBASE-6466 add the possibility to have multiple threads for the flusher. > The default value is 1, but this should be incremented to 2 reasons: > - I've observed that the flush of a region can be delayed because another is > in progress. During a write load, this leads to an increased latency because > the memstore size increases and then block the client > - if, by accident, a flusher hits a slow or bad datanode, all the flush will > have to wait until the timeouts expires. With 2 or more flushers and some > luck the other regions will be flushed by the second thread. > Lastly this setting is important enough to be documented in the standard > hbase site imho. > I think it's important enough to go in the .98 branch as well > There will be an impact: as the flush won't be queued (or less queued) we may > have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007997#comment-14007997 ] Hudson commented on HBASE-11226: ABORTED: Integrated in HBase-0.98 #313 (See [https://builds.apache.org/job/HBase-0.98/313/]) HBASE-11226 Document and increase the default value for hbase.hstore.flusher.count (nkeywal: rev 996705c61de1c7b533024dcb41cc571531ebfbb9) * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java > Document and increase the default value for hbase.hstore.flusher.count > -- > > Key: HBASE-11226 > URL: https://issues.apache.org/jira/browse/HBASE-11226 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.99.0, 0.98.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.99.0, 0.98.3 > > Attachments: 11226.v1.patch > > > HBASE-6466 add the possibility to have multiple threads for the flusher. > The default value is 1, but this should be incremented to 2 reasons: > - I've observed that the flush of a region can be delayed because another is > in progress. During a write load, this leads to an increased latency because > the memstore size increases and then block the client > - if, by accident, a flusher hits a slow or bad datanode, all the flush will > have to wait until the timeouts expires. With 2 or more flushers and some > luck the other regions will be flushed by the second thread. > Lastly this setting is important enough to be documented in the standard > hbase site imho. > I think it's important enough to go in the .98 branch as well > There will be an impact: as the flush won't be queued (or less queued) we may > have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14007470#comment-14007470 ] Hudson commented on HBASE-11226: FAILURE: Integrated in HBase-TRUNK #5139 (See [https://builds.apache.org/job/HBase-TRUNK/5139/]) HBASE-11226 Document and increase the default value for hbase.hstore.flusher.count (nkeywal: rev 28adf6065888b0f88fd252cd32ee1bb1725baaee) * hbase-common/src/main/resources/hbase-default.xml * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java > Document and increase the default value for hbase.hstore.flusher.count > -- > > Key: HBASE-11226 > URL: https://issues.apache.org/jira/browse/HBASE-11226 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.99.0, 0.98.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.99.0, 0.98.3 > > Attachments: 11226.v1.patch > > > HBASE-6466 add the possibility to have multiple threads for the flusher. > The default value is 1, but this should be incremented to 2 reasons: > - I've observed that the flush of a region can be delayed because another is > in progress. During a write load, this leads to an increased latency because > the memstore size increases and then block the client > - if, by accident, a flusher hits a slow or bad datanode, all the flush will > have to wait until the timeouts expires. With 2 or more flushers and some > luck the other regions will be flushed by the second thread. > Lastly this setting is important enough to be documented in the standard > hbase site imho. > I think it's important enough to go in the .98 branch as well > There will be an impact: as the flush won't be queued (or less queued) we may > have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14005122#comment-14005122 ] Nick Dimiduk commented on HBASE-11226: -- ACK. +1 > Document and increase the default value for hbase.hstore.flusher.count > -- > > Key: HBASE-11226 > URL: https://issues.apache.org/jira/browse/HBASE-11226 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.99.0, 0.98.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.99.0, 0.98.3 > > Attachments: 11226.v1.patch > > > HBASE-6466 add the possibility to have multiple threads for the flusher. > The default value is 1, but this should be incremented to 2 reasons: > - I've observed that the flush of a region can be delayed because another is > in progress. During a write load, this leads to an increased latency because > the memstore size increases and then block the client > - if, by accident, a flusher hits a slow or bad datanode, all the flush will > have to wait until the timeouts expires. With 2 or more flushers and some > luck the other regions will be flushed by the second thread. > Lastly this setting is important enough to be documented in the standard > hbase site imho. > I think it's important enough to go in the .98 branch as well > There will be an impact: as the flush won't be queued (or less queued) we may > have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004922#comment-14004922 ] Nicolas Liochon commented on HBASE-11226: - yes and no. No, because in theory there is no queue, so changing this setting changes nothing, except if one datanode is having issue. Yes, because in practice I've seen this. There is a limit on the number of files. HBASE-11208 is about removing it, but this is too much of a change for a 0.98 default imho... You can see that the queuing is happening in the logs: if the actual flush size is much larger than the configured flush size, it means that you got queued. > Document and increase the default value for hbase.hstore.flusher.count > -- > > Key: HBASE-11226 > URL: https://issues.apache.org/jira/browse/HBASE-11226 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.99.0, 0.98.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.99.0, 0.98.3 > > Attachments: 11226.v1.patch > > > HBASE-6466 add the possibility to have multiple threads for the flusher. > The default value is 1, but this should be incremented to 2 reasons: > - I've observed that the flush of a region can be delayed because another is > in progress. During a write load, this leads to an increased latency because > the memstore size increases and then block the client > - if, by accident, a flusher hits a slow or bad datanode, all the flush will > have to wait until the timeouts expires. With 2 or more flushers and some > luck the other regions will be flushed by the second thread. > Lastly this setting is important enough to be documented in the standard > hbase site imho. > I think it's important enough to go in the .98 branch as well > There will be an impact: as the flush won't be queued (or less queued) we may > have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004903#comment-14004903 ] Nick Dimiduk commented on HBASE-11226: -- As I understand it, 2 flusher threads will mean more frequent flushes and thus more, smaller files. Should this be accompanied with an increase to hbase.hstore.blockingStoreFiles ? Basically, it moves the blocking pressure from memstore global limit to compaction queue? > Document and increase the default value for hbase.hstore.flusher.count > -- > > Key: HBASE-11226 > URL: https://issues.apache.org/jira/browse/HBASE-11226 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.99.0, 0.98.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.99.0, 0.98.3 > > Attachments: 11226.v1.patch > > > HBASE-6466 add the possibility to have multiple threads for the flusher. > The default value is 1, but this should be incremented to 2 reasons: > - I've observed that the flush of a region can be delayed because another is > in progress. During a write load, this leads to an increased latency because > the memstore size increases and then block the client > - if, by accident, a flusher hits a slow or bad datanode, all the flush will > have to wait until the timeouts expires. With 2 or more flushers and some > luck the other regions will be flushed by the second thread. > Lastly this setting is important enough to be documented in the standard > hbase site imho. > I think it's important enough to go in the .98 branch as well > There will be an impact: as the flush won't be queued (or less queued) we may > have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11226) Document and increase the default value for hbase.hstore.flusher.count
[ https://issues.apache.org/jira/browse/HBASE-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004887#comment-14004887 ] Andrew Purtell commented on HBASE-11226: +1 for 0.98. > Document and increase the default value for hbase.hstore.flusher.count > -- > > Key: HBASE-11226 > URL: https://issues.apache.org/jira/browse/HBASE-11226 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.99.0, 0.98.2 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon > Fix For: 0.99.0, 0.98.3 > > > HBASE-6466 add the possibility to have multiple threads for the flusher. > The default value is 1, but this should be incremented to 2 reasons: > - I've observed that the flush of a region can be delayed because another is > in progress. During a write load, this leads to an increased latency because > the memstore size increases and then block the client > - if, by accident, a flusher hits a slow or bad datanode, all the flush will > have to wait until the timeouts expires. With 2 or more flushers and some > luck the other regions will be flushed by the second thread. > Lastly this setting is important enough to be documented in the standard > hbase site imho. > I think it's important enough to go in the .98 branch as well > There will be an impact: as the flush won't be queued (or less queued) we may > have more compactions... -- This message was sent by Atlassian JIRA (v6.2#6252)
