[jira] [Updated] (HBASE-11643) Read and write MOB in HBase
[ https://issues.apache.org/jira/browse/HBASE-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11643: - Attachment: HBASE-11643-V3.diff Rename the mob threshold in the constants. And adjust the line width in the code to less than or equal with 100. > Read and write MOB in HBase > --- > > Key: HBASE-11643 > URL: https://issues.apache.org/jira/browse/HBASE-11643 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-11643-V2.diff, HBASE-11643-V3.diff, > HBase-11643.diff > > > The read/write MOB in HBase are implemented in this JIRA. Normally, the Cells > are saved in the MemStore, and flushed to the HFiles when necessary. For MOB, > the Cells are saved in the MemStore as well, but they're flushed to elsewhere > out of HBase in the format of HFiles. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11695) PeriodicFlusher and WakeFrequency issues
[
https://issues.apache.org/jira/browse/HBASE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl resolved HBASE-11695.
---
Resolution: Invalid
Not worth fixing.
> PeriodicFlusher and WakeFrequency issues
>
>
> Key: HBASE-11695
> URL: https://issues.apache.org/jira/browse/HBASE-11695
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.21
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Attachments: 11695-trunk.txt
>
>
> We just ran into a flush storm caused by the PeriodicFlusher.
> Many memstore became eligible for flushing at exactly the same time, the
> effect we've seen is that the exact same region was flushed multiple times,
> because the flusher wakes up too often (every 10s). The jitter of 20s is
> larger than that and it takes some time to actually flush the memstore.
> Here's one example. We've seen 100's of these, monopolizing the flush queue
> and preventing "important" flushes from happening.
> {code}
> 06-Aug-2014 20:11:56 [regionserver60020.periodicFlusher] INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer[1397]--
> regionserver60020.periodicFlusher requesting flush for region
> tsdb,\x00\x00\x0AO\xCF*
> \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2.
> after a delay of 13449
> 06-Aug-2014 20:12:06 [regionserver60020.periodicFlusher] INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer[1397]--
> regionserver60020.periodicFlusher requesting flush for region
> tsdb,\x00\x00\x0AO\xCF*
> \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2.
> after a delay of 14060
> {code}
> So we need to increase the period of the PeriodicFlusher to at least the
> random jitter, also increase the default random jitter (20s does not help
> with many regions).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11695) PeriodicFlusher and WakeFrequency issues
[
https://issues.apache.org/jira/browse/HBASE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-11695:
--
Fix Version/s: (was: 0.98.6)
(was: 0.94.23)
(was: 2.0.0)
(was: 0.99.0)
> PeriodicFlusher and WakeFrequency issues
>
>
> Key: HBASE-11695
> URL: https://issues.apache.org/jira/browse/HBASE-11695
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.21
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Attachments: 11695-trunk.txt
>
>
> We just ran into a flush storm caused by the PeriodicFlusher.
> Many memstore became eligible for flushing at exactly the same time, the
> effect we've seen is that the exact same region was flushed multiple times,
> because the flusher wakes up too often (every 10s). The jitter of 20s is
> larger than that and it takes some time to actually flush the memstore.
> Here's one example. We've seen 100's of these, monopolizing the flush queue
> and preventing "important" flushes from happening.
> {code}
> 06-Aug-2014 20:11:56 [regionserver60020.periodicFlusher] INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer[1397]--
> regionserver60020.periodicFlusher requesting flush for region
> tsdb,\x00\x00\x0AO\xCF*
> \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2.
> after a delay of 13449
> 06-Aug-2014 20:12:06 [regionserver60020.periodicFlusher] INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer[1397]--
> regionserver60020.periodicFlusher requesting flush for region
> tsdb,\x00\x00\x0AO\xCF*
> \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2.
> after a delay of 14060
> {code}
> So we need to increase the period of the PeriodicFlusher to at least the
> random jitter, also increase the default random jitter (20s does not help
> with many regions).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11059) ZK-less region assignment
[ https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090372#comment-14090372 ] ramkrishna.s.vasudevan commented on HBASE-11059: Thanks Jimmy. Read further down the code. > ZK-less region assignment > - > > Key: HBASE-11059 > URL: https://issues.apache.org/jira/browse/HBASE-11059 > Project: HBase > Issue Type: Improvement > Components: master, Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.99.0 > > Attachments: hbase-11059.patch, hbase-11059_v2.1.patch, > hbase-11059_v2.2.patch, hbase-11059_v2.patch, hbase-11059_v3.0.patch, > zk-less_am.pdf, zk-less_assignment.png > > > It seems that most people don't like region assignment with ZK (HBASE-5487), > which causes many uncertainties. This jira is to support ZK-less region > assignment. We need to make sure this patch doesn't break backward > compatibility/rolling upgrade. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11696) Make CombinedBlockCache resizable.
[ https://issues.apache.org/jira/browse/HBASE-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090363#comment-14090363 ] Anoop Sam John commented on HBASE-11696: It depends on the auto tuner impl. The default simple one will increase size (till max size) when there are evictions- until to reach a non evicting state. Also this it can do only when there is very less/no write load and there are no blocking flushes due to global heap pressure. And regarding DATA/META blocks hosting, will it be better we have separate cache parts for META and DATA blocks? Not gone into more details in this area. > Make CombinedBlockCache resizable. > -- > > Key: HBASE-11696 > URL: https://issues.apache.org/jira/browse/HBASE-11696 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > > HBASE-5349 adds auto tuning of memstore heap and block cache heap. Block > cache needs to be resizable in order for this. CombinedBlockCache is not > marked resizable now. We can make this. On resize the L1 cache (ie. LRU > cache) can get resized. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11699) Region servers exclusion list to HMaster.
[ https://issues.apache.org/jira/browse/HBASE-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090348#comment-14090348 ] Gomathivinayagam Muthuvinayagam commented on HBASE-11699: - Thanks for quick response, Few questions 1) Can I disable the balancer while I shutdown region server, suppose if a region server that is added to exclusion list which is running currently? Is it recommended? or adding few additional parameters to the configuration which informs the administrators that balancer may be disabled while a region server is shutdown by the new code? 2) Currently, graceful shutdown uses a ruby script to unload regions from a region server before shutting down that region server. I am just planning to convert that code to java equivalent code. What is your comment on this one? 3) Can I add api in the HBaseAdmin to add/remove region servers to exclusion list? > Region servers exclusion list to HMaster. > - > > Key: HBASE-11699 > URL: https://issues.apache.org/jira/browse/HBASE-11699 > Project: HBase > Issue Type: New Feature > Components: Admin, Client, regionserver, Zookeeper >Affects Versions: 0.98.3 >Reporter: Gomathivinayagam Muthuvinayagam >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Original Estimate: 96h > Remaining Estimate: 96h > > Currently HBase does not support adding set of region servers to be in the > exclusion list. So that administrators can prevent accidental startups of > some region servers to join the cluster. There was initially some work done, > and it is available in https://issues.apache.org/jira/browse/HBASE-3833. It > was not done after that. > I am planning to contribute it as a patch, and I would like to do some > improvements as well. Instead of storing the exclusion entries on a file, I > am planning to store it on zookeeper. Can anyone suggest thoughts on this? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11643) Read and write MOB in HBase
[ https://issues.apache.org/jira/browse/HBASE-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11643: - Attachment: HBASE-11643-V2.diff Sync the code from the master and generate the patch. Re-attach it to pass the Hadoop QA > Read and write MOB in HBase > --- > > Key: HBASE-11643 > URL: https://issues.apache.org/jira/browse/HBASE-11643 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Jingcheng Du >Assignee: Jingcheng Du > Attachments: HBASE-11643-V2.diff, HBase-11643.diff > > > The read/write MOB in HBase are implemented in this JIRA. Normally, the Cells > are saved in the MemStore, and flushed to the HFiles when necessary. For MOB, > the Cells are saved in the MemStore as well, but they're flushed to elsewhere > out of HBase in the format of HFiles. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11581) Add option so CombinedBlockCache L2 can be null (fscache)
[ https://issues.apache.org/jira/browse/HBASE-11581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11581: -- Labels: beginner beginners (was: ) > Add option so CombinedBlockCache L2 can be null (fscache) > - > > Key: HBASE-11581 > URL: https://issues.apache.org/jira/browse/HBASE-11581 > Project: HBase > Issue Type: New Feature > Components: BlockCache >Reporter: stack >Assignee: stack >Priority: Minor > Labels: beginner, beginners > Attachments: 11581.txt > > > Add option, mostly for comparison's sake, that allows a deploy orchestrated > by CombinedBlockCache such that its L1 is LruBlockCache for META blocks but > DATA blocks are fetched each time (we don't try and cache them, no blockcache > churn). > In operation, i can see fscache coming around to cover the fetched DATA > blocks such that if the DATA blocks fit in fscache, seeks go to zero. > This setup for sure runs slower. Will publish numbers elsewhere. Meantime, > here is a patch to enable this option. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11581) Add option so CombinedBlockCache L2 can be null (fscache)
[
https://issues.apache.org/jira/browse/HBASE-11581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090340#comment-14090340
]
stack commented on HBASE-11581:
---
Thanks [~jmhsieh] for review.
A better way to do this, a more 'natural' way is fixing the 'Cache DATA on
Read' flag. Currently it is overloaded, compounded with the BLOCKCACHE =>
true/false flag. It needs to be made distinct. I just tried on a table
setting CF to false but BC got deployed anyways.
{code}
hbase(main):004:0> enable 'TestTable'
0 row(s) in 0.8470 seconds
hbase(main):005:0> describe 'TestTable'
DESCRIPTION
ENABLED
'TestTable', {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER =>
'ROW', REPLICATION_SCOPE => '0', VERSIONS => '2', COMP true
RESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS
=> 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false
', BLOCKCACHE => 'false'}
1 row(s) in 0.0430 seconds
hbase(main):006:0>
hbase(main):007:0*
hbase(main):008:0* disable 'TestTable'
0 row(s) in 1.2930 seconds
hbase(main):009:0> describe 'TestTable'
DESCRIPTION
ENABLED
'TestTable', {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER =>
'ROW', REPLICATION_SCOPE => '0', VERSIONS => '2', COMP false
RESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS
=> 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false
', BLOCKCACHE => 'false'}
1 row(s) in 0.0340 seconds
hbase(main):010:0> alter 'TestTable', {NAME => 'info', BLOCKCACHE => false}
Updating all regions with the new schema...
16/16 regions updated.
Done.
0 row(s) in 1.1950 seconds
hbase(main):011:0> describe 'TestTable'
DESCRIPTION
ENABLED
'TestTable', {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER =>
'ROW', REPLICATION_SCOPE => '0', VERSIONS => '2', COMP false
RESSION => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS
=> 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false
', BLOCKCACHE => 'false'}
1 row(s) in 0.0330 seconds
hbase(main):012:0> enable 'TestTable'
0 row(s) in 0.7850 seconds
{code}
> Add option so CombinedBlockCache L2 can be null (fscache)
> -
>
> Key: HBASE-11581
> URL: https://issues.apache.org/jira/browse/HBASE-11581
> Project: HBase
> Issue Type: New Feature
> Components: BlockCache
>Reporter: stack
>Assignee: stack
>Priority: Minor
> Labels: beginner, beginners
> Attachments: 11581.txt
>
>
> Add option, mostly for comparison's sake, that allows a deploy orchestrated
> by CombinedBlockCache such that its L1 is LruBlockCache for META blocks but
> DATA blocks are fetched each time (we don't try and cache them, no blockcache
> churn).
> In operation, i can see fscache coming around to cover the fetched DATA
> blocks such that if the DATA blocks fit in fscache, seeks go to zero.
> This setup for sure runs slower. Will publish numbers elsewhere. Meantime,
> here is a patch to enable this option.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-4593) Design and document the official procedure for posting patches, commits, commit messages, etc. to smooth process and make integration with tools easier
[
https://issues.apache.org/jira/browse/HBASE-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090324#comment-14090324
]
Sean Busbey commented on HBASE-4593:
{quote}
+
+Jira
+Check for existing issues in https://issues.apache.org/jira/browse/HBASE";>Jira. If it's
+either a new feature request, enhancement, or a bug, file a
ticket.
+To check for existing issues which you can tackle as a
beginner, search for https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20labels%20in%20(beginner)"
+>issues in JIRA tagged with the label 'beginner'.
See for more information.
{quote}
Now that this is 18.1.3, referring back to 18.1 doesn't make much sense.
{quote}
Maven Build Commands
@@ -1061,18 +1317,18 @@ batch.restart.rs.ratio=0.4f
Note: use Maven 3 (Maven 2 may work but we suggest you use Maven
3).
-
+
Compile
mvn compile
-
+
-
+
Running all or individual Unit Tests
See the section
above in
-
+
Building against various hadoop versions.
{quote}
With the current layout, it looks like 18.10.1 can be moved under 18.4. With it
there, the intro to 18.10 can be dropped. Also the reference to 18.10 that's in
18.4.1 (esp since there is more info about different maven commands in 18.4
than 18.10).
{quote}
+If you need to revise your patch, leave the previous
patch file(s)
+attached to the JIRA, and upload the new one, as in
. Cancel the
Patch Available flag
+and then re-trigger it, by toggling the
Patch
+Available button in JIRA. JIRA sorts
attached files by the
+time they were attached, and has no problem with
multiple attachments with
+the same name.
+
+
+If you need to submit yuor patch against multiple
branches, rather than
+just master, name each version of the patch with the
branch it is for, as in
+.
{quote}
the links to "submitting.patches.naming" show up as ???. Are list items
reference-able or is this a rendering problem?
{quote}
+
Space Invaders
-Rather than do this...
+Do not use extra spaces around brackets. Use the second
style, rather than the
+first.
{quote}
Why do these start as examples and then turn into sections? For example, the
example on autogenerated code is similar in scope and presentation as the
section for useless javadocs.
We should make them all one or the other. I'd favor sections, but we're already
deeply nested.
{quote}
+
+Implementing Writable
+
+Applies pre-0.96 only
+In 0.96, HBase moved to protocol buffers
(protobufs). The below section on
+Writables applies to 0.94.x and previous, not to 0.96
and beyond.
+
+Every class returned by RegionServers must implement the
Writable
+interface. If you are creating a new class that needs to
implement this
+interface, do not forget the default constructor.
+
{quote}
Since this example is only for some branches, it should probably go after all
the other guidelines.
{quote}
+Review Group. If you fill in
the
+Bugs field, the review board
is attached to the
+relevant JIRA automatically. The more fields you fill
in, the better. Click
{quote}
I don't think this is true. At the very least I've been having to attach them
manually (see HBASE-11658). Filling in the bugs field will automatically
provide a link from the reviewboard to the jira, but not the reverse.
{quote}
+
()
+HBASE-12345 Fix All The Things
(j...@example.com)
+If the submitter used git
format-patch to generate the
+patch, their commit message is in their patch and you
can use that.
{quote}
Refer to contributor instead of submitter.
A warning about checking that the contributor properly started their message
with a jira-id would be good.
> Design and document the official procedure for posting patches, commits,
> commit messages, etc. to smooth process and make integration with tools easier
> ---
>
>
[jira] [Assigned] (HBASE-11405) Multiple invocations of hbck in parallel disables balancer permanently
[
https://issues.apache.org/jira/browse/HBASE-11405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bharath v reassigned HBASE-11405:
-
Assignee: bharath v
> Multiple invocations of hbck in parallel disables balancer permanently
> ---
>
> Key: HBASE-11405
> URL: https://issues.apache.org/jira/browse/HBASE-11405
> Project: HBase
> Issue Type: Bug
> Components: Balancer, hbck
>Affects Versions: 0.99.0
>Reporter: bharath v
>Assignee: bharath v
> Attachments: HBASE-11405-trunk.patch, HBASE-11405-trunk.patch.1
>
>
> This is because of the following piece of code in hbck
> {code:borderStyle=solid}
> boolean oldBalancer = admin.setBalancerRunning(false, true);
> try {
> onlineConsistencyRepair();
> }
> finally {
> admin.setBalancerRunning(oldBalancer, false);
> }
> {code}
> Newer invocations set oldBalancer to false as it was disabled by previous
> invocations and this disables balancer permanently unless its manually turned
> on by the user. Easy to reproduce, just run hbck 100 times in a loop in 2
> different sessions and you can see that balancer is set to false in the
> HMaster logs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11701) Start and end of memstore flush log should be on the same level
[ https://issues.apache.org/jira/browse/HBASE-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090312#comment-14090312 ] Hudson commented on HBASE-11701: SUCCESS: Integrated in HBase-0.98 #442 (See [https://builds.apache.org/job/HBase-0.98/442/]) HBASE-11701 Start and end of memstore flush log should be on the same level (Esteban Gutierrez) (apurtell: rev 80c1e6724d90bb74af34238ce6336ced05ec) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Start and end of memstore flush log should be on the same level > --- > > Key: HBASE-11701 > URL: https://issues.apache.org/jira/browse/HBASE-11701 > Project: HBase > Issue Type: Improvement > Components: Usability >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11701.patch > > > Currently the start of the memstore flush is done in DEBUG level and when it > finishes is done in INFO. Both should be done in the same level. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11323) BucketCache all the time!
[ https://issues.apache.org/jira/browse/HBASE-11323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-11323. --- Resolution: Fixed Release Note: Use the LruBlockCache default if your data fits the blockcache. If block cache churn or you want a block cache that is immune to the vagaries of BC, deploy the offheap bucketcache. See http://people.apache.org/~stack/bc/ > BucketCache all the time! > - > > Key: HBASE-11323 > URL: https://issues.apache.org/jira/browse/HBASE-11323 > Project: HBase > Issue Type: Sub-task > Components: io >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: > BlockCacheReportLruBlockCachevsOffHeapCombinedBlockCacheSmall4G (1).pdf, > ReportBlockCache.pdf > > > One way to realize the parent issue is to just enable bucket cache all the > time; i.e. always have offheap enabled. Would have to do some work to make > it drop-dead simple on initial setup (I think it doable). > So, upside would be the offheap upsides (less GC, less likely to go away and > never come back because of full GC when heap is large, etc.). > Downside is higher latency. In Nick's BlockCache 101 there is little to no > difference between onheap and offheap. In a basic compare doing scans and > gets -- details to follow -- I have BucketCache deploy about 20% less ops > than LRUBC when all incache and maybe 10% less ops when falling out of cache. > I can't tell difference in means and 95th and 99th are roughly same (more > stable with BucketCache). GC profile is much better with BucketCache -- way > less. BucketCache uses about 7% more user CPU. > More detail on comparison to follow. > I think the numbers disagree enough we should probably do the [~lhofhansl] > suggestion, that we allow you to have a table sit in LRUBC, something the > current bucket cache layout does not do. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11693) Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090308#comment-14090308 ] Hudson commented on HBASE-11693: SUCCESS: Integrated in HBase-0.94 #1400 (See [https://builds.apache.org/job/HBase-0.94/1400/]) HBASE-11693 Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94 (apurtell: rev 18ce81375d875461431c8868600301a6304f48b1) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * src/test/java/org/apache/hadoop/hbase/filter/FilterAllFilter.java > Backport HBASE-11026 (Provide option to filter out all rows in > PerformanceEvaluation tool) to 0.94 > -- > > Key: HBASE-11693 > URL: https://issues.apache.org/jira/browse/HBASE-11693 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11693.patch > > > Backport HBASE-11026 (Provide option to filter out all rows in > PerformanceEvaluation tool) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11026) Provide option to filter out all rows in PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090305#comment-14090305 ] Hudson commented on HBASE-11026: FAILURE: Integrated in HBase-0.94-JDK7 #169 (See [https://builds.apache.org/job/HBase-0.94-JDK7/169/]) HBASE-11693 Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94 (apurtell: rev 18ce81375d875461431c8868600301a6304f48b1) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * src/test/java/org/apache/hadoop/hbase/filter/FilterAllFilter.java > Provide option to filter out all rows in PerformanceEvaluation tool > --- > > Key: HBASE-11026 > URL: https://issues.apache.org/jira/browse/HBASE-11026 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.99.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11026_1.patch, HBASE-11026_2.patch, > HBASE-11026_4-0.98.patch, HBASE-11026_4.patch > > > Performance Evaluation could also be used to check the actual performance of > the scans on the Server side by passing Filters that filters out all the > rows. We can create a test filter and add it to the Filter.proto and set > this filter based on input params. Could be helpful in testing. > If you feel this is not needed pls feel free to close this issue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11026) Provide option to filter out all rows in PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090307#comment-14090307 ] Hudson commented on HBASE-11026: SUCCESS: Integrated in HBase-0.94 #1400 (See [https://builds.apache.org/job/HBase-0.94/1400/]) HBASE-11693 Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94 (apurtell: rev 18ce81375d875461431c8868600301a6304f48b1) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * src/test/java/org/apache/hadoop/hbase/filter/FilterAllFilter.java > Provide option to filter out all rows in PerformanceEvaluation tool > --- > > Key: HBASE-11026 > URL: https://issues.apache.org/jira/browse/HBASE-11026 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.99.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11026_1.patch, HBASE-11026_2.patch, > HBASE-11026_4-0.98.patch, HBASE-11026_4.patch > > > Performance Evaluation could also be used to check the actual performance of > the scans on the Server side by passing Filters that filters out all the > rows. We can create a test filter and add it to the Filter.proto and set > this filter based on input params. Could be helpful in testing. > If you feel this is not needed pls feel free to close this issue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11693) Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090306#comment-14090306 ] Hudson commented on HBASE-11693: FAILURE: Integrated in HBase-0.94-JDK7 #169 (See [https://builds.apache.org/job/HBase-0.94-JDK7/169/]) HBASE-11693 Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94 (apurtell: rev 18ce81375d875461431c8868600301a6304f48b1) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java * src/test/java/org/apache/hadoop/hbase/filter/FilterAllFilter.java > Backport HBASE-11026 (Provide option to filter out all rows in > PerformanceEvaluation tool) to 0.94 > -- > > Key: HBASE-11693 > URL: https://issues.apache.org/jira/browse/HBASE-11693 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11693.patch > > > Backport HBASE-11026 (Provide option to filter out all rows in > PerformanceEvaluation tool) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11701) Start and end of memstore flush log should be on the same level
[ https://issues.apache.org/jira/browse/HBASE-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090301#comment-14090301 ] Hudson commented on HBASE-11701: SUCCESS: Integrated in HBase-TRUNK #5383 (See [https://builds.apache.org/job/HBase-TRUNK/5383/]) HBASE-11701 Start and end of memstore flush log should be on the same level (Esteban Gutierrez) (apurtell: rev da01d2f53d0dbb2c1ad6eb79a481058285d91419) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Start and end of memstore flush log should be on the same level > --- > > Key: HBASE-11701 > URL: https://issues.apache.org/jira/browse/HBASE-11701 > Project: HBase > Issue Type: Improvement > Components: Usability >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11701.patch > > > Currently the start of the memstore flush is done in DEBUG level and when it > finishes is done in INFO. Both should be done in the same level. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11682) Explain hotspotting
[ https://issues.apache.org/jira/browse/HBASE-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11682: Attachment: HBASE-11682.patch Let me know how this version suits you guys. Thanks for the feedback. > Explain hotspotting > --- > > Key: HBASE-11682 > URL: https://issues.apache.org/jira/browse/HBASE-11682 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11682-1.patch, HBASE-11682.patch, HBASE-11682.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11059) ZK-less region assignment
[ https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090291#comment-14090291 ] Jimmy Xiang commented on HBASE-11059: - The method name is a little confusing. This method doesn't update meta directly. It sends a RPC call to master to update the meta instead. > ZK-less region assignment > - > > Key: HBASE-11059 > URL: https://issues.apache.org/jira/browse/HBASE-11059 > Project: HBase > Issue Type: Improvement > Components: master, Region Assignment >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.99.0 > > Attachments: hbase-11059.patch, hbase-11059_v2.1.patch, > hbase-11059_v2.2.patch, hbase-11059_v2.patch, hbase-11059_v3.0.patch, > zk-less_am.pdf, zk-less_assignment.png > > > It seems that most people don't like region assignment with ZK (HBASE-5487), > which causes many uncertainties. This jira is to support ZK-less region > assignment. We need to make sure this patch doesn't break backward > compatibility/rolling upgrade. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11682) Explain hotspotting
[ https://issues.apache.org/jira/browse/HBASE-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11682: Status: Patch Available (was: Open) > Explain hotspotting > --- > > Key: HBASE-11682 > URL: https://issues.apache.org/jira/browse/HBASE-11682 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-11682-1.patch, HBASE-11682.patch, HBASE-11682.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11059) ZK-less region assignment
[
https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090286#comment-14090286
]
ramkrishna.s.vasudevan commented on HBASE-11059:
[~jxiang]
The doc suggests that only the master updates the meta. In the master branch
the HMaster itself is am HRS.
In the OpenRegionHandler we still do updateMeta(). So is that done by the
RegionServer or the master?
{code}
if (!updateMeta(region) || this.server.isStopped() ||
this.rsServices.isStopping()) {
return;
}
{code}
> ZK-less region assignment
> -
>
> Key: HBASE-11059
> URL: https://issues.apache.org/jira/browse/HBASE-11059
> Project: HBase
> Issue Type: Improvement
> Components: master, Region Assignment
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.99.0
>
> Attachments: hbase-11059.patch, hbase-11059_v2.1.patch,
> hbase-11059_v2.2.patch, hbase-11059_v2.patch, hbase-11059_v3.0.patch,
> zk-less_am.pdf, zk-less_assignment.png
>
>
> It seems that most people don't like region assignment with ZK (HBASE-5487),
> which causes many uncertainties. This jira is to support ZK-less region
> assignment. We need to make sure this patch doesn't break backward
> compatibility/rolling upgrade.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11673) TestIOFencing#testFencingAroundCompactionAfterWALSync fails
[
https://issues.apache.org/jira/browse/HBASE-11673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090283#comment-14090283
]
Mikhail Antonov commented on HBASE-11673:
-
[~tedyu]
looking at https://builds.apache.org/view/All/job/HBase-TRUNK/5381/console
results, I'm seeing this test passed:
{quote}
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.378 sec
Running org.apache.hadoop.hbase.TestMultiVersions
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.35 sec
Running org.apache.hadoop.hbase.TestIOFencing
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 195.001 sec
Running org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.649 sec
{quote}
Do you think this issue could be resolved now?
> TestIOFencing#testFencingAroundCompactionAfterWALSync fails
> ---
>
> Key: HBASE-11673
> URL: https://issues.apache.org/jira/browse/HBASE-11673
> Project: HBase
> Issue Type: Test
>Reporter: Qiang Tian
>Assignee: Sergey Soldatov
> Fix For: 2.0.0
>
> Attachments: HBASE_11673-v1.patch,
> testFencingAroundCompactionAfterWALSync.tar.gz
>
>
> got several test failure on the latest build:
> {quote}
> [tianq@bdvm101 surefire-reports]$ ls -1t|grep "Tests run" * |grep "<<<
> FAILURE"
> org.apache.hadoop.hbase.client.TestReplicasClient.txt:Tests run: 1, Failures:
> 0, Errors: 1, Skipped: 0, Time elapsed: 38.706 sec <<< FAILURE!
> org.apache.hadoop.hbase.master.TestMasterOperationsForRegionReplicas.txt:Tests
> run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 30.669 sec <<<
> FAILURE!
> org.apache.hadoop.hbase.regionserver.TestRegionReplicas.txt:Tests run: 1,
> Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 39.113 sec <<< FAILURE!
> org.apache.hadoop.hbase.TestIOFencing.txt:Tests run: 2, Failures: 1, Errors:
> 0, Skipped: 0, Time elapsed: 177.071 sec <<< FAILURE!
> {quote}
> the first one:
> {quote}
> type="java.lang.AssertionError">java.lang.AssertionError: Timed out waiting
> for the region to flush
> >-at org.junit.Assert.fail(Assert.java:88)
> >-at org.junit.Assert.assertTrue(Assert.java:41)
> >-at org.apache.hadoop.hbase.TestIOFencing.doTest(TestIOFencing.java:291)
> >-at
> >org.apache.hadoop.hbase.TestIOFencing.testFencingAroundCompactionAfterWALSync(TestIOFencing.java:236)
> >-at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >-at
> >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >-at
> >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >-at java.lang.reflect.Method.invoke(Method.java:606)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11701) Start and end of memstore flush log should be on the same level
[ https://issues.apache.org/jira/browse/HBASE-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090271#comment-14090271 ] Hudson commented on HBASE-11701: FAILURE: Integrated in HBase-1.0 #91 (See [https://builds.apache.org/job/HBase-1.0/91/]) HBASE-11701 Start and end of memstore flush log should be on the same level (Esteban Gutierrez) (apurtell: rev 5dd2afd6708c42b9d0f043a8da9ce540af26afef) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Start and end of memstore flush log should be on the same level > --- > > Key: HBASE-11701 > URL: https://issues.apache.org/jira/browse/HBASE-11701 > Project: HBase > Issue Type: Improvement > Components: Usability >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11701.patch > > > Currently the start of the memstore flush is done in DEBUG level and when it > finishes is done in INFO. Both should be done in the same level. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11696) Make CombinedBlockCache resizable.
[ https://issues.apache.org/jira/browse/HBASE-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090257#comment-14090257 ] stack commented on HBASE-11696: --- Yes. You are right. If the flag is set on a CF to store DATA blocks in L1, it will be done. What you thinking? We'll resize L1 until no evictions up to a max size (whether hosting DATA and/or META)? > Make CombinedBlockCache resizable. > -- > > Key: HBASE-11696 > URL: https://issues.apache.org/jira/browse/HBASE-11696 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > > HBASE-5349 adds auto tuning of memstore heap and block cache heap. Block > cache needs to be resizable in order for this. CombinedBlockCache is not > marked resizable now. We can make this. On resize the L1 cache (ie. LRU > cache) can get resized. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11697) Improve the 'Too many blocks' message on UI blockcache status page
[ https://issues.apache.org/jira/browse/HBASE-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090255#comment-14090255 ] stack commented on HBASE-11697: --- We are listing out buckets of bucketcache. If too many, we only list the first portion. I think we need to flag that we are not showing all. > Improve the 'Too many blocks' message on UI blockcache status page > -- > > Key: HBASE-11697 > URL: https://issues.apache.org/jira/browse/HBASE-11697 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11697.patch, TooManyBlocks.png > > > If metrics calculations over blockcache contents stopped after examining > hbase.ui.blockcache.by.file.max items, the UI will put up a message. However, > this notion of "too many blocks" / fullness refers to structures used for > calculating blockcache metrics. See BlockCacheUtil. We should improve this > message so it does not leave a user the impression the blockcache may be in a > bad state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11026) Provide option to filter out all rows in PerformanceEvaluation tool
[ https://issues.apache.org/jira/browse/HBASE-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090238#comment-14090238 ] Hudson commented on HBASE-11026: FAILURE: Integrated in HBase-0.94-security #510 (See [https://builds.apache.org/job/HBase-0.94-security/510/]) HBASE-11693 Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94 (apurtell: rev 18ce81375d875461431c8868600301a6304f48b1) * src/test/java/org/apache/hadoop/hbase/filter/FilterAllFilter.java * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Provide option to filter out all rows in PerformanceEvaluation tool > --- > > Key: HBASE-11026 > URL: https://issues.apache.org/jira/browse/HBASE-11026 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.99.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11026_1.patch, HBASE-11026_2.patch, > HBASE-11026_4-0.98.patch, HBASE-11026_4.patch > > > Performance Evaluation could also be used to check the actual performance of > the scans on the Server side by passing Filters that filters out all the > rows. We can create a test filter and add it to the Filter.proto and set > this filter based on input params. Could be helpful in testing. > If you feel this is not needed pls feel free to close this issue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-5934) Add the ability for Performance Evaluation to set the table compression
[ https://issues.apache.org/jira/browse/HBASE-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090240#comment-14090240 ] Hudson commented on HBASE-5934: --- FAILURE: Integrated in HBase-0.94-security #510 (See [https://builds.apache.org/job/HBase-0.94-security/510/]) HBASE-11690 Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94 (apurtell: rev 46c26949ca5aa7b1a34f06778b058f1e3ad72364) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Add the ability for Performance Evaluation to set the table compression > --- > > Key: HBASE-5934 > URL: https://issues.apache.org/jira/browse/HBASE-5934 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.95.0 > > Attachments: HBASE-5934-v0.patch, HBASE-5934-v1.patch > > > When testing it's nice to get a more realistic set of numbers. As such > allowing the tool to create pre-split regions was needed. Compression should > be the same. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7156) Add Data Block Encoding and -D opts to Performance Evaluation
[ https://issues.apache.org/jira/browse/HBASE-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090243#comment-14090243 ] Hudson commented on HBASE-7156: --- FAILURE: Integrated in HBase-0.94-security #510 (See [https://builds.apache.org/job/HBase-0.94-security/510/]) HBASE-11691 Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94 (apurtell: rev 9a8142d11f39f37a24486f77ac989a483643a1ba) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Add Data Block Encoding and -D opts to Performance Evaluation > - > > Key: HBASE-7156 > URL: https://issues.apache.org/jira/browse/HBASE-7156 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.95.0 > > Attachments: HBASE-7156-v0.patch, HBASE-7156-v1.patch, > HBASE-7156-v2.patch, HBASE-7156-v3.patch > > > Add the ability to specify Data Block Encoding and other configuration > options. > --blockEncoding=TYPE > -D > Example: > hbase org.apache.hadoop.hbase.PerformanceEvaluation -D > mapreduce.task.timeout=6 --blockEncoding=DIFF sequentialWrite 1 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11693) Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090239#comment-14090239 ] Hudson commented on HBASE-11693: FAILURE: Integrated in HBase-0.94-security #510 (See [https://builds.apache.org/job/HBase-0.94-security/510/]) HBASE-11693 Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94 (apurtell: rev 18ce81375d875461431c8868600301a6304f48b1) * src/test/java/org/apache/hadoop/hbase/filter/FilterAllFilter.java * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Backport HBASE-11026 (Provide option to filter out all rows in > PerformanceEvaluation tool) to 0.94 > -- > > Key: HBASE-11693 > URL: https://issues.apache.org/jira/browse/HBASE-11693 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11693.patch > > > Backport HBASE-11026 (Provide option to filter out all rows in > PerformanceEvaluation tool) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11690) Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090241#comment-14090241 ] Hudson commented on HBASE-11690: FAILURE: Integrated in HBase-0.94-security #510 (See [https://builds.apache.org/job/HBase-0.94-security/510/]) HBASE-11690 Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94 (apurtell: rev 46c26949ca5aa7b1a34f06778b058f1e3ad72364) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Backport HBASE-5934 (Add the ability for Performance Evaluation to set the > table compression) to 0.94 > - > > Key: HBASE-11690 > URL: https://issues.apache.org/jira/browse/HBASE-11690 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.23 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11690.patch > > > Backport HBASE-5934 (Add the ability for Performance Evaluation to set the > table compression) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11691) Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090242#comment-14090242 ] Hudson commented on HBASE-11691: FAILURE: Integrated in HBase-0.94-security #510 (See [https://builds.apache.org/job/HBase-0.94-security/510/]) HBASE-11691 Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94 (apurtell: rev 9a8142d11f39f37a24486f77ac989a483643a1ba) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance > Evaluation) to 0.94 > --- > > Key: HBASE-11691 > URL: https://issues.apache.org/jira/browse/HBASE-11691 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11691.patch > > > Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance > Evaluation) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11697) Improve the 'Too many blocks' message on UI blockcache status page
[ https://issues.apache.org/jira/browse/HBASE-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090219#comment-14090219 ] Hudson commented on HBASE-11697: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #415 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/415/]) HBASE-11697 Improve the 'Too many blocks' message on UI blockcache status page (Mikhail Antonov) (apurtell: rev 6283ec730596b2e47666e8bef3a0fdb0442f55e6) * hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon > Improve the 'Too many blocks' message on UI blockcache status page > -- > > Key: HBASE-11697 > URL: https://issues.apache.org/jira/browse/HBASE-11697 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11697.patch, TooManyBlocks.png > > > If metrics calculations over blockcache contents stopped after examining > hbase.ui.blockcache.by.file.max items, the UI will put up a message. However, > this notion of "too many blocks" / fullness refers to structures used for > calculating blockcache metrics. See BlockCacheUtil. We should improve this > message so it does not leave a user the impression the blockcache may be in a > bad state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090218#comment-14090218 ] Hudson commented on HBASE-11678: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #415 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/415/]) HBASE-11678 BucketCache ramCache fills heap after running a few hours (stack: rev a41eca43e34dee7570e94a4f62080b4b541fbe7a) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketWriterThread.java * hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestProtoBufRpc.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java HBASE-11678 BucketCache ramCache fills heap after running a few hours -- 0.98 addendum (stack: rev b8e7c61645404d50fb247307ea590f71a6a7472e) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, 11678v3.098.txt, 11678v3.txt, 11678v3.txt, > gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11508) Document changes to IPC config parameters from HBASE-11492
[ https://issues.apache.org/jira/browse/HBASE-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11508: Status: Patch Available (was: Open) > Document changes to IPC config parameters from HBASE-11492 > -- > > Key: HBASE-11508 > URL: https://issues.apache.org/jira/browse/HBASE-11508 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-11492.patch, HBASE-11508-1.patch, HBASE-11508.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11508) Document changes to IPC config parameters from HBASE-11492
[ https://issues.apache.org/jira/browse/HBASE-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11508: Attachment: HBASE-11508.patch Clarified version info. > Document changes to IPC config parameters from HBASE-11492 > -- > > Key: HBASE-11508 > URL: https://issues.apache.org/jira/browse/HBASE-11508 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-11492.patch, HBASE-11508-1.patch, HBASE-11508.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11508) Document changes to IPC config parameters from HBASE-11492
[ https://issues.apache.org/jira/browse/HBASE-11508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11508: Status: Open (was: Patch Available) > Document changes to IPC config parameters from HBASE-11492 > -- > > Key: HBASE-11508 > URL: https://issues.apache.org/jira/browse/HBASE-11508 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-11492.patch, HBASE-11508-1.patch, HBASE-11508.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Issue Comment Deleted] (HBASE-11696) Make CombinedBlockCache resizable.
[
https://issues.apache.org/jira/browse/HBASE-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-11696:
---
Comment: was deleted
(was: Yes [~stack] I got it like META blocks u r referring.
{code}
/**
* @param value true if we should cache data blocks in the L1 cache (if block
cache deploy
* has more than one tier; e.g. we are using CombinedBlockCache).
* @return this (for chained invocation)
*/
public HColumnDescriptor setCacheDataInL1(boolean value) {
return setValue(CACHE_DATA_IN_L1, Boolean.toString(value));
}
{code}
The above javadoc confused me. I thought the data blocks for that HCD will be
stored in L1.)
> Make CombinedBlockCache resizable.
> --
>
> Key: HBASE-11696
> URL: https://issues.apache.org/jira/browse/HBASE-11696
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
>
> HBASE-5349 adds auto tuning of memstore heap and block cache heap. Block
> cache needs to be resizable in order for this. CombinedBlockCache is not
> marked resizable now. We can make this. On resize the L1 cache (ie. LRU
> cache) can get resized.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11696) Make CombinedBlockCache resizable.
[
https://issues.apache.org/jira/browse/HBASE-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090212#comment-14090212
]
Anoop Sam John commented on HBASE-11696:
Yes [~stack] I got it like META blocks u r referring.
{code}
/**
* @param value true if we should cache data blocks in the L1 cache (if block
cache deploy
* has more than one tier; e.g. we are using CombinedBlockCache).
* @return this (for chained invocation)
*/
public HColumnDescriptor setCacheDataInL1(boolean value) {
return setValue(CACHE_DATA_IN_L1, Boolean.toString(value));
}
{code}
The above javadoc confused me. I thought the data blocks for that HCD will be
stored in L1.
> Make CombinedBlockCache resizable.
> --
>
> Key: HBASE-11696
> URL: https://issues.apache.org/jira/browse/HBASE-11696
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
>
> HBASE-5349 adds auto tuning of memstore heap and block cache heap. Block
> cache needs to be resizable in order for this. CombinedBlockCache is not
> marked resizable now. We can make this. On resize the L1 cache (ie. LRU
> cache) can get resized.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11696) Make CombinedBlockCache resizable.
[
https://issues.apache.org/jira/browse/HBASE-11696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090213#comment-14090213
]
Anoop Sam John commented on HBASE-11696:
Yes [~stack] I got it like META blocks u r referring.
{code}
/**
* @param value true if we should cache data blocks in the L1 cache (if block
cache deploy
* has more than one tier; e.g. we are using CombinedBlockCache).
* @return this (for chained invocation)
*/
public HColumnDescriptor setCacheDataInL1(boolean value) {
return setValue(CACHE_DATA_IN_L1, Boolean.toString(value));
}
{code}
The above javadoc confused me. I thought the data blocks for that HCD will be
stored in L1.
> Make CombinedBlockCache resizable.
> --
>
> Key: HBASE-11696
> URL: https://issues.apache.org/jira/browse/HBASE-11696
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
>
> HBASE-5349 adds auto tuning of memstore heap and block cache heap. Block
> cache needs to be resizable in order for this. CombinedBlockCache is not
> marked resizable now. We can make this. On resize the L1 cache (ie. LRU
> cache) can get resized.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11553) Abstract visibility label related services into an interface
[ https://issues.apache.org/jira/browse/HBASE-11553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090210#comment-14090210 ] Anoop Sam John commented on HBASE-11553: Yes [~apurtell] the RB is already updated with latest patch. > Abstract visibility label related services into an interface > > > Key: HBASE-11553 > URL: https://issues.apache.org/jira/browse/HBASE-11553 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11553.patch, HBASE-11553.patch, > HBASE-11553_V2.patch, HBASE-11553_V3.patch > > > - storage and retrieval of label dictionary and authentication sets > - marshalling and unmarshalling of visibility expression representations > in operation attributes and cell tags > - management of assignment of authorizations to principals > This will allow us to introduce additional serde implementations for > visibility expressions, for example storing as strings in some places and > compressed/tokenized representation in others in order to support additional > use cases. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-4593) Design and document the official procedure for posting patches, commits, commit messages, etc. to smooth process and make integration with tools easier
[ https://issues.apache.org/jira/browse/HBASE-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-4593: --- Attachment: (was: HBASE-4593.pdf) > Design and document the official procedure for posting patches, commits, > commit messages, etc. to smooth process and make integration with tools easier > --- > > Key: HBASE-4593 > URL: https://issues.apache.org/jira/browse/HBASE-4593 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Jonathan Gray >Assignee: Misty Stanley-Jones > Attachments: Chapter_18_and_19.pdf, HBASE-4593.patch, HBASE-4593.patch > > > I have been building a tool (currently called reposync) to help me keep the > internal FB hbase-92-based branch up-to-date with the public branches. > Various inconsistencies in our process has made it difficult to automate a > lot of this stuff. > I'd like to work with everyone to come up with the official best practices > and stick to it. > I welcome all suggestions. Among some of the things I'd like to nail down: > - Commit message format > - Best practice and commit message format for multiple commits > - Multiple commits per jira vs. jira per commit, what are the exceptions and > when > - Affects vs. Fix versions > - Potential usage of [tags] in commit messages for things like book, scripts, > shell... maybe even whatever is in the components field? > - Increased usage of JIRA tags or labels to mark exactly which repos a JIRA > has been committed to (potentially even internal repos? ways for a tool to > keep track in JIRA?) > We also need to be more strict about some things if we want to follow Apache > guidelines. For example, all final versions of a patch must be attached to > JIRA so that the author properly assigns it to Apache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-4593) Design and document the official procedure for posting patches, commits, commit messages, etc. to smooth process and make integration with tools easier
[ https://issues.apache.org/jira/browse/HBASE-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-4593: --- Attachment: Chapter_18_and_19.pdf This PDF render should be better as it comes from Acrobat. It includes chapter 18 and the new chapter 19, where I broke out the unit testing bit that has to do with testing your own application. > Design and document the official procedure for posting patches, commits, > commit messages, etc. to smooth process and make integration with tools easier > --- > > Key: HBASE-4593 > URL: https://issues.apache.org/jira/browse/HBASE-4593 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Jonathan Gray >Assignee: Misty Stanley-Jones > Attachments: Chapter_18_and_19.pdf, HBASE-4593.patch, HBASE-4593.patch > > > I have been building a tool (currently called reposync) to help me keep the > internal FB hbase-92-based branch up-to-date with the public branches. > Various inconsistencies in our process has made it difficult to automate a > lot of this stuff. > I'd like to work with everyone to come up with the official best practices > and stick to it. > I welcome all suggestions. Among some of the things I'd like to nail down: > - Commit message format > - Best practice and commit message format for multiple commits > - Multiple commits per jira vs. jira per commit, what are the exceptions and > when > - Affects vs. Fix versions > - Potential usage of [tags] in commit messages for things like book, scripts, > shell... maybe even whatever is in the components field? > - Increased usage of JIRA tags or labels to mark exactly which repos a JIRA > has been committed to (potentially even internal repos? ways for a tool to > keep track in JIRA?) > We also need to be more strict about some things if we want to follow Apache > guidelines. For example, all final versions of a patch must be attached to > JIRA so that the author properly assigns it to Apache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11700) Fix typo of hbase.store.compaction.ratio (should be hbase.hstore.compaction.ratio) in Ref Guide
[ https://issues.apache.org/jira/browse/HBASE-11700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones resolved HBASE-11700. - Resolution: Duplicate > Fix typo of hbase.store.compaction.ratio (should be > hbase.hstore.compaction.ratio) in Ref Guide > --- > > Key: HBASE-11700 > URL: https://issues.apache.org/jira/browse/HBASE-11700 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Misty Stanley-Jones > Labels: beginner > > Hey, noticed that in the doc it lists out the following option: > hbase.store.compaction.ratio but it should be hbase.hstore.compaction.ratio. > Doc looks really good though! I will hopefully have more feedback for you > soon. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11685) Incr/decr on the reference count of HConnectionImplementation need be atomic
[
https://issues.apache.org/jira/browse/HBASE-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-11685:
Attachment: HBASE-11685-trunk-v4.diff
> Incr/decr on the reference count of HConnectionImplementation need be atomic
> -
>
> Key: HBASE-11685
> URL: https://issues.apache.org/jira/browse/HBASE-11685
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-11685-trunk-v1.diff, HBASE-11685-trunk-v2.diff,
> HBASE-11685-trunk-v3.diff, HBASE-11685-trunk-v4.diff
>
>
> Currently, the incr/decr operation on the ref count of
> HConnectionImplementation are not atomic. This may cause that the ref count
> always be larger than 0 and the connection never be closed.
> {code}
> /**
> * Increment this client's reference count.
> */
> void incCount() {
> ++refCount;
> }
> /**
> * Decrement this client's reference count.
> */
> void decCount() {
> if (refCount > 0) {
> --refCount;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11685) Incr/decr on the reference count of HConnectionImplementation need be atomic
[
https://issues.apache.org/jira/browse/HBASE-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-11685:
Attachment: HBASE-11685-trunk-v3.diff
Fix the typo and add connection name in warn log for debug.
Thanks for [~apurtell]'s suggestion.
> Incr/decr on the reference count of HConnectionImplementation need be atomic
> -
>
> Key: HBASE-11685
> URL: https://issues.apache.org/jira/browse/HBASE-11685
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-11685-trunk-v1.diff, HBASE-11685-trunk-v2.diff,
> HBASE-11685-trunk-v3.diff
>
>
> Currently, the incr/decr operation on the ref count of
> HConnectionImplementation are not atomic. This may cause that the ref count
> always be larger than 0 and the connection never be closed.
> {code}
> /**
> * Increment this client's reference count.
> */
> void incCount() {
> ++refCount;
> }
> /**
> * Decrement this client's reference count.
> */
> void decCount() {
> if (refCount > 0) {
> --refCount;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11685) Incr/decr on the reference count of HConnectionImplementation need be atomic
[
https://issues.apache.org/jira/browse/HBASE-11685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-11685:
Attachment: (was: HBASE-11685-trunk-v3.diff)
> Incr/decr on the reference count of HConnectionImplementation need be atomic
> -
>
> Key: HBASE-11685
> URL: https://issues.apache.org/jira/browse/HBASE-11685
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-11685-trunk-v1.diff, HBASE-11685-trunk-v2.diff,
> HBASE-11685-trunk-v3.diff
>
>
> Currently, the incr/decr operation on the ref count of
> HConnectionImplementation are not atomic. This may cause that the ref count
> always be larger than 0 and the connection never be closed.
> {code}
> /**
> * Increment this client's reference count.
> */
> void incCount() {
> ++refCount;
> }
> /**
> * Decrement this client's reference count.
> */
> void decCount() {
> if (refCount > 0) {
> --refCount;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11700) Fix typo of hbase.store.compaction.ratio (should be hbase.hstore.compaction.ratio) in Ref Guide
[ https://issues.apache.org/jira/browse/HBASE-11700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090179#comment-14090179 ] Liu Shaohui commented on HBASE-11700: - [~misty] Has been Fixed in HBASE-11648. Thanks for your report. > Fix typo of hbase.store.compaction.ratio (should be > hbase.hstore.compaction.ratio) in Ref Guide > --- > > Key: HBASE-11700 > URL: https://issues.apache.org/jira/browse/HBASE-11700 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Misty Stanley-Jones > Labels: beginner > > Hey, noticed that in the doc it lists out the following option: > hbase.store.compaction.ratio but it should be hbase.hstore.compaction.ratio. > Doc looks really good though! I will hopefully have more feedback for you > soon. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-4593) Design and document the official procedure for posting patches, commits, commit messages, etc. to smooth process and make integration with tools easier
[ https://issues.apache.org/jira/browse/HBASE-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-4593: --- Attachment: HBASE-4593.patch Addressed feedback from [~busbey] and [~apurtell]. > Design and document the official procedure for posting patches, commits, > commit messages, etc. to smooth process and make integration with tools easier > --- > > Key: HBASE-4593 > URL: https://issues.apache.org/jira/browse/HBASE-4593 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Jonathan Gray >Assignee: Misty Stanley-Jones > Attachments: HBASE-4593.patch, HBASE-4593.patch, HBASE-4593.pdf > > > I have been building a tool (currently called reposync) to help me keep the > internal FB hbase-92-based branch up-to-date with the public branches. > Various inconsistencies in our process has made it difficult to automate a > lot of this stuff. > I'd like to work with everyone to come up with the official best practices > and stick to it. > I welcome all suggestions. Among some of the things I'd like to nail down: > - Commit message format > - Best practice and commit message format for multiple commits > - Multiple commits per jira vs. jira per commit, what are the exceptions and > when > - Affects vs. Fix versions > - Potential usage of [tags] in commit messages for things like book, scripts, > shell... maybe even whatever is in the components field? > - Increased usage of JIRA tags or labels to mark exactly which repos a JIRA > has been committed to (potentially even internal repos? ways for a tool to > keep track in JIRA?) > We also need to be more strict about some things if we want to follow Apache > guidelines. For example, all final versions of a patch must be attached to > JIRA so that the author properly assigns it to Apache. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11691) Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090176#comment-14090176 ] Hudson commented on HBASE-11691: FAILURE: Integrated in HBase-0.94-JDK7 #168 (See [https://builds.apache.org/job/HBase-0.94-JDK7/168/]) HBASE-11691 Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94 (apurtell: rev 9a8142d11f39f37a24486f77ac989a483643a1ba) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance > Evaluation) to 0.94 > --- > > Key: HBASE-11691 > URL: https://issues.apache.org/jira/browse/HBASE-11691 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11691.patch > > > Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance > Evaluation) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-5934) Add the ability for Performance Evaluation to set the table compression
[ https://issues.apache.org/jira/browse/HBASE-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090174#comment-14090174 ] Hudson commented on HBASE-5934: --- FAILURE: Integrated in HBase-0.94-JDK7 #168 (See [https://builds.apache.org/job/HBase-0.94-JDK7/168/]) HBASE-11690 Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94 (apurtell: rev 46c26949ca5aa7b1a34f06778b058f1e3ad72364) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Add the ability for Performance Evaluation to set the table compression > --- > > Key: HBASE-5934 > URL: https://issues.apache.org/jira/browse/HBASE-5934 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.95.0 > > Attachments: HBASE-5934-v0.patch, HBASE-5934-v1.patch > > > When testing it's nice to get a more realistic set of numbers. As such > allowing the tool to create pre-split regions was needed. Compression should > be the same. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7156) Add Data Block Encoding and -D opts to Performance Evaluation
[ https://issues.apache.org/jira/browse/HBASE-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090177#comment-14090177 ] Hudson commented on HBASE-7156: --- FAILURE: Integrated in HBase-0.94-JDK7 #168 (See [https://builds.apache.org/job/HBase-0.94-JDK7/168/]) HBASE-11691 Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94 (apurtell: rev 9a8142d11f39f37a24486f77ac989a483643a1ba) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Add Data Block Encoding and -D opts to Performance Evaluation > - > > Key: HBASE-7156 > URL: https://issues.apache.org/jira/browse/HBASE-7156 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.95.0 > > Attachments: HBASE-7156-v0.patch, HBASE-7156-v1.patch, > HBASE-7156-v2.patch, HBASE-7156-v3.patch > > > Add the ability to specify Data Block Encoding and other configuration > options. > --blockEncoding=TYPE > -D > Example: > hbase org.apache.hadoop.hbase.PerformanceEvaluation -D > mapreduce.task.timeout=6 --blockEncoding=DIFF sequentialWrite 1 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11690) Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090175#comment-14090175 ] Hudson commented on HBASE-11690: FAILURE: Integrated in HBase-0.94-JDK7 #168 (See [https://builds.apache.org/job/HBase-0.94-JDK7/168/]) HBASE-11690 Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94 (apurtell: rev 46c26949ca5aa7b1a34f06778b058f1e3ad72364) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Backport HBASE-5934 (Add the ability for Performance Evaluation to set the > table compression) to 0.94 > - > > Key: HBASE-11690 > URL: https://issues.apache.org/jira/browse/HBASE-11690 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.23 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11690.patch > > > Backport HBASE-5934 (Add the ability for Performance Evaluation to set the > table compression) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-7156) Add Data Block Encoding and -D opts to Performance Evaluation
[ https://issues.apache.org/jira/browse/HBASE-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090171#comment-14090171 ] Hudson commented on HBASE-7156: --- FAILURE: Integrated in HBase-0.94 #1399 (See [https://builds.apache.org/job/HBase-0.94/1399/]) HBASE-11691 Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94 (apurtell: rev 9a8142d11f39f37a24486f77ac989a483643a1ba) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Add Data Block Encoding and -D opts to Performance Evaluation > - > > Key: HBASE-7156 > URL: https://issues.apache.org/jira/browse/HBASE-7156 > Project: HBase > Issue Type: Improvement > Components: test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.95.0 > > Attachments: HBASE-7156-v0.patch, HBASE-7156-v1.patch, > HBASE-7156-v2.patch, HBASE-7156-v3.patch > > > Add the ability to specify Data Block Encoding and other configuration > options. > --blockEncoding=TYPE > -D > Example: > hbase org.apache.hadoop.hbase.PerformanceEvaluation -D > mapreduce.task.timeout=6 --blockEncoding=DIFF sequentialWrite 1 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11690) Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090169#comment-14090169 ] Hudson commented on HBASE-11690: FAILURE: Integrated in HBase-0.94 #1399 (See [https://builds.apache.org/job/HBase-0.94/1399/]) HBASE-11690 Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94 (apurtell: rev 46c26949ca5aa7b1a34f06778b058f1e3ad72364) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Backport HBASE-5934 (Add the ability for Performance Evaluation to set the > table compression) to 0.94 > - > > Key: HBASE-11690 > URL: https://issues.apache.org/jira/browse/HBASE-11690 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.23 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11690.patch > > > Backport HBASE-5934 (Add the ability for Performance Evaluation to set the > table compression) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11691) Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090170#comment-14090170 ] Hudson commented on HBASE-11691: FAILURE: Integrated in HBase-0.94 #1399 (See [https://builds.apache.org/job/HBase-0.94/1399/]) HBASE-11691 Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94 (apurtell: rev 9a8142d11f39f37a24486f77ac989a483643a1ba) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance > Evaluation) to 0.94 > --- > > Key: HBASE-11691 > URL: https://issues.apache.org/jira/browse/HBASE-11691 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11691.patch > > > Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance > Evaluation) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-5934) Add the ability for Performance Evaluation to set the table compression
[ https://issues.apache.org/jira/browse/HBASE-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090168#comment-14090168 ] Hudson commented on HBASE-5934: --- FAILURE: Integrated in HBase-0.94 #1399 (See [https://builds.apache.org/job/HBase-0.94/1399/]) HBASE-11690 Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94 (apurtell: rev 46c26949ca5aa7b1a34f06778b058f1e3ad72364) * src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java > Add the ability for Performance Evaluation to set the table compression > --- > > Key: HBASE-5934 > URL: https://issues.apache.org/jira/browse/HBASE-5934 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.95.0 > > Attachments: HBASE-5934-v0.patch, HBASE-5934-v1.patch > > > When testing it's nice to get a more realistic set of numbers. As such > allowing the tool to create pre-split regions was needed. Compression should > be the same. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-4593) Design and document the official procedure for posting patches, commits, commit messages, etc. to smooth process and make integration with tools easier
[
https://issues.apache.org/jira/browse/HBASE-4593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090151#comment-14090151
]
Andrew Purtell commented on HBASE-4593:
---
Fantastic draft!
In addition to Sean's comments above:
---
In 18.4. Building Apache HBase, we should also describe a developer can build
themselves a tarball without going through the whole release procedure. I use
{noformat}
mvn -DskipTests clean install && mvn -DskipTests package assembly:single
{noformat}
The distribution tarball can be found at
{{hbase-assembly/target/hbase--bin.tar.gz}}
---
Procedure 18.2. Release Procedure, step 1 says "Update the CHANGES.txt file"
but this is also about updating POMs as well as CHANGES.txt.
---
Elsewhere in the release procedure instructions, the 'MAVEN_OPTS="-Xmx3g"'
appearing in example commands is an artifact of a Mac OS X environment. This
isn't needed on Linux in my experience. We could remove the MAVEN_OPTS here and
put in a footnote instead.
---
In 18.12.1. Create Patch, "If you need to revise your patch, leave the previous
patch file(s) attached to the JIRA, and upload the new one with a revision ID."
This isn't strictly necessary. JIRA will sort files with the same name
descending by date.
---
18.12.2. Unit Tests
Perhaps after this add a section 'Integration Tests'.
"Significant new features should provide an integration test in addition to
unit tests, suitable for exercising the new feature at different points in its
configuration space." Or something like that?
---
Will make another pass later. So much material.
> Design and document the official procedure for posting patches, commits,
> commit messages, etc. to smooth process and make integration with tools easier
> ---
>
> Key: HBASE-4593
> URL: https://issues.apache.org/jira/browse/HBASE-4593
> Project: HBase
> Issue Type: Task
> Components: documentation
>Reporter: Jonathan Gray
>Assignee: Misty Stanley-Jones
> Attachments: HBASE-4593.patch, HBASE-4593.pdf
>
>
> I have been building a tool (currently called reposync) to help me keep the
> internal FB hbase-92-based branch up-to-date with the public branches.
> Various inconsistencies in our process has made it difficult to automate a
> lot of this stuff.
> I'd like to work with everyone to come up with the official best practices
> and stick to it.
> I welcome all suggestions. Among some of the things I'd like to nail down:
> - Commit message format
> - Best practice and commit message format for multiple commits
> - Multiple commits per jira vs. jira per commit, what are the exceptions and
> when
> - Affects vs. Fix versions
> - Potential usage of [tags] in commit messages for things like book, scripts,
> shell... maybe even whatever is in the components field?
> - Increased usage of JIRA tags or labels to mark exactly which repos a JIRA
> has been committed to (potentially even internal repos? ways for a tool to
> keep track in JIRA?)
> We also need to be more strict about some things if we want to follow Apache
> guidelines. For example, all final versions of a patch must be attached to
> JIRA so that the author properly assigns it to Apache.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11704) Include set-up instructions for building with IDEs besides Eclipse
[ https://issues.apache.org/jira/browse/HBASE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090141#comment-14090141 ] Misty Stanley-Jones commented on HBASE-11704: - There is other stuff in there like importing the formatter. Do we have that for other IDEs too? > Include set-up instructions for building with IDEs besides Eclipse > -- > > Key: HBASE-11704 > URL: https://issues.apache.org/jira/browse/HBASE-11704 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Misty Stanley-Jones > > We have pretty detailed instructions for setting up Eclipse to build HBase at > http://hbase.apache.org/book/ides.html#eclipse. It would be nice to have this > for other popular IDEs such as IntelliJ. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11704) Include set-up instructions for building with IDEs besides Eclipse
[ https://issues.apache.org/jira/browse/HBASE-11704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090138#comment-14090138 ] Nick Dimiduk commented on HBASE-11704: -- I don't think there's anything special that needs to be done on any IDE anymore. It's just a maven project. Pretty standard stuff. Do we know the eclipse content is still necessary? I never found I needed it, but I've been off eclipse for over a year. How about just a note saying "import the maven project per your tool's instructions" and leave it at that. Less is more, right? > Include set-up instructions for building with IDEs besides Eclipse > -- > > Key: HBASE-11704 > URL: https://issues.apache.org/jira/browse/HBASE-11704 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Misty Stanley-Jones > > We have pretty detailed instructions for setting up Eclipse to build HBase at > http://hbase.apache.org/book/ides.html#eclipse. It would be nice to have this > for other popular IDEs such as IntelliJ. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11699) Region servers exclusion list to HMaster.
[ https://issues.apache.org/jira/browse/HBASE-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11699: --- Fix Version/s: (was: 0.98.3) 0.98.6 2.0.0 0.99.0 Set fix versions for upcoming release > Region servers exclusion list to HMaster. > - > > Key: HBASE-11699 > URL: https://issues.apache.org/jira/browse/HBASE-11699 > Project: HBase > Issue Type: New Feature > Components: Admin, Client, regionserver, Zookeeper >Affects Versions: 0.98.3 >Reporter: Gomathivinayagam Muthuvinayagam >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Original Estimate: 96h > Remaining Estimate: 96h > > Currently HBase does not support adding set of region servers to be in the > exclusion list. So that administrators can prevent accidental startups of > some region servers to join the cluster. There was initially some work done, > and it is available in https://issues.apache.org/jira/browse/HBASE-3833. It > was not done after that. > I am planning to contribute it as a patch, and I would like to do some > improvements as well. Instead of storing the exclusion entries on a file, I > am planning to store it on zookeeper. Can anyone suggest thoughts on this? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11701) Start and end of memstore flush log should be on the same level
[ https://issues.apache.org/jira/browse/HBASE-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11701: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to 0.98+ > Start and end of memstore flush log should be on the same level > --- > > Key: HBASE-11701 > URL: https://issues.apache.org/jira/browse/HBASE-11701 > Project: HBase > Issue Type: Improvement > Components: Usability >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11701.patch > > > Currently the start of the memstore flush is done in DEBUG level and when it > finishes is done in INFO. Both should be done in the same level. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11701) Start and end of memstore flush log should be on the same level
[
https://issues.apache.org/jira/browse/HBASE-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090120#comment-14090120
]
Hadoop QA commented on HBASE-11701:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660495/HBASE-11701.patch
against trunk revision .
ATTACHMENT ID: 12660495
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10342//console
This message is automatically generated.
> Start and end of memstore flush log should be on the same level
> ---
>
> Key: HBASE-11701
> URL: https://issues.apache.org/jira/browse/HBASE-11701
> Project: HBase
> Issue Type: Improvement
> Components: Usability
>Reporter: Esteban Gutierrez
>Assignee: Esteban Gutierrez
>Priority: Trivial
> Fix For: 0.99.0, 2.0.0, 0.98.6
>
> Attachments: HBASE-11701.patch
>
>
> Currently the start of the memstore flush is done in DEBUG level and when it
> finishes is done in INFO. Both should be done in the same level.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-11693) Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-11693. Resolution: Fixed Hadoop Flags: Reviewed Pushed to 0.94 > Backport HBASE-11026 (Provide option to filter out all rows in > PerformanceEvaluation tool) to 0.94 > -- > > Key: HBASE-11693 > URL: https://issues.apache.org/jira/browse/HBASE-11693 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11693.patch > > > Backport HBASE-11026 (Provide option to filter out all rows in > PerformanceEvaluation tool) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11693) Backport HBASE-11026 (Provide option to filter out all rows in PerformanceEvaluation tool) to 0.94
[
https://issues.apache.org/jira/browse/HBASE-11693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090114#comment-14090114
]
Andrew Purtell commented on HBASE-11693:
Set up a single node HDFS+HBase+YARN cluster.
{noformat}
$ ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1
[...]
$ ./bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --filterAll scan 5
[...]
2014-08-07 17:49:43,370 INFO [main] mapreduce.Job: Job job_1407458412239_0001
completed successfully
{noformat}
Application logs show '--filterAll' taking effect.
> Backport HBASE-11026 (Provide option to filter out all rows in
> PerformanceEvaluation tool) to 0.94
> --
>
> Key: HBASE-11693
> URL: https://issues.apache.org/jira/browse/HBASE-11693
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Minor
> Fix For: 0.94.23
>
> Attachments: HBASE-11693.patch
>
>
> Backport HBASE-11026 (Provide option to filter out all rows in
> PerformanceEvaluation tool) to 0.94 branch.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-6626) Add a chapter on HDFS in the troubleshooting section of the HBase reference guide.
[ https://issues.apache.org/jira/browse/HBASE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090109#comment-14090109 ] Hudson commented on HBASE-6626: --- FAILURE: Integrated in HBase-TRUNK #5382 (See [https://builds.apache.org/job/HBase-TRUNK/5382/]) HBASE-6626 Add a chapter on HDFS in the troubleshooting section of the HBase reference guide (Misty Stanley-Jones) (jmhsieh: rev 5848710aa707c41d1c3ecaa1442859ab2b9f878c) * src/main/docbkx/troubleshooting.xml > Add a chapter on HDFS in the troubleshooting section of the HBase reference > guide. > -- > > Key: HBASE-6626 > URL: https://issues.apache.org/jira/browse/HBASE-6626 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.95.2 >Reporter: Nicolas Liochon >Assignee: Misty Stanley-Jones >Priority: Blocker > Fix For: 0.99.0, 2.0.0 > > Attachments: HBASE-6626.patch, troubleshooting.txt > > > I looked mainly at the major failure case, but here is what I have: > New sub chapter in the existing chapter "Troubleshooting and Debugging > HBase": "HDFS & HBASE" > 1) HDFS & HBase > 2) Connection related settings > 2.1) Number of retries > 2.2) Timeouts > 3) Log samples > 1) HDFS & HBase > HBase uses HDFS to store its HFile, i.e. the core HBase files and the > Write-Ahead-Logs, i.e. the files that will be used to restore the data after > a crash. > In both cases, the reliability of HBase comes from the fact that HDFS writes > the data to multiple locations. To be efficient, HBase needs the data to be > available locally, hence it's highly recommended to have the HDFS datanode on > the same machines as the HBase Region Servers. > Detailed information on how HDFS works can be found at [1]. > Important features are: > - HBase is a client application of HDFS, i.e. uses the HDFS DFSClient class. > This class can appears in HBase logs with other HDFS client related logs. > - Some HDFS settings are HDFS-server-side, i.e. must be set on the HDFS > side, while some other are HDFS-client-side, i.e. must be set in HBase, while > some other must be set in both places. > - the HDFS writes are pipelined from one datanode to another. When writing, > there are communications between: > - HBase and HDFS namenode, through the HDFS client classes. > - HBase and HDFS datanodes, through the HDFS client classes. > - HDFS datanode between themselves: issues on these communications are in > HDFS logs, not HBase. HDFS writes are always local when possible. As a > consequence, there should not be much write error in HBase Region Servers: > they write to the local datanode. If this datanode can't replicate the > blocks, it will appear in its logs, not in the region servers logs. > - datanodes can be contacted through the ipc.Client interface (once again > this class can shows up in HBase logs) and the data transfer interface > (usually shows up as the DataNode class in the HBase logs). There are on > different ports (defaults being: 50010 and 50020). > - To understand exactly what's going on, you must look that the HDFS log > files as well: HBase logs represent the client side. > - With the default setting, HDFS needs 630s to mark a datanode as dead. For > this reason, this node will still be tried by HBase or by other datanodes > when writing and reading until HDFS definitively decides it's dead. This will > add some extras lines in the logs. This monitoring is performed by the > NameNode. > - The HDFS clients (i.e. HBase using HDFS client code) don't fully rely on > the NameNode, but can mark temporally a node as dead if they had an error > when they tried to use it. > 2) Settings for retries and timeouts > 2.1) Retries > ipc.client.connect.max.retries > Default 10 > Indicates the number of retries a client will make to establish a server > connection. Not taken into account if the error is a SocketTimeout. In this > case the number of retries is 45 (fixed on branch, HADOOP-7932 or in > HADOOP-7397). For SASL, the number of retries is hard-coded to 15. Can be > increased, especially if the socket timeouts have been lowered. > ipc.client.connect.max.retries.on.timeouts > Default 45 > If you have HADOOP-7932, max number of retries on timeout. Counter is > different than ipc.client.connect.max.retries so if you mix the socket errors > you will get 55 retries with the default values. Could be lowered, once it is > available. With HADOOP-7397 ipc.client.connect.max.retries is reused so there > would be 10 tries. > dfs.client.block.write.retries > Default 3 > Number of tries for the client when writing a block. After a failure, will > connect to the namenode a get a new location, sending the list of the > datanodes alread
[jira] [Commented] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090111#comment-14090111 ] Hudson commented on HBASE-11678: FAILURE: Integrated in HBase-TRUNK #5382 (See [https://builds.apache.org/job/HBase-TRUNK/5382/]) HBASE-11678 BucketCache ramCache fills heap after running a few hours (stack: rev 5e899c68e053d2c07733a89297d5056511e2e802) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketWriterThread.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/HeapMemorySizeUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java * hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestProtoBufRpc.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, 11678v3.098.txt, 11678v3.txt, 11678v3.txt, > gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11697) Improve the 'Too many blocks' message on UI blockcache status page
[ https://issues.apache.org/jira/browse/HBASE-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090112#comment-14090112 ] Hudson commented on HBASE-11697: FAILURE: Integrated in HBase-TRUNK #5382 (See [https://builds.apache.org/job/HBase-TRUNK/5382/]) HBASE-11697 Improve the 'Too many blocks' message on UI blockcache status page (Mikhail Antonov) (apurtell: rev e11e89e5afc55d1e1256e85138e84f8a25b4ed96) * hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon > Improve the 'Too many blocks' message on UI blockcache status page > -- > > Key: HBASE-11697 > URL: https://issues.apache.org/jira/browse/HBASE-11697 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11697.patch, TooManyBlocks.png > > > If metrics calculations over blockcache contents stopped after examining > hbase.ui.blockcache.by.file.max items, the UI will put up a message. However, > this notion of "too many blocks" / fullness refers to structures used for > calculating blockcache metrics. See BlockCacheUtil. We should improve this > message so it does not leave a user the impression the blockcache may be in a > bad state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11333) Remove deprecated class MetaMigrationConvertingToPB
[ https://issues.apache.org/jira/browse/HBASE-11333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090110#comment-14090110 ] Hudson commented on HBASE-11333: FAILURE: Integrated in HBase-TRUNK #5382 (See [https://builds.apache.org/job/HBase-TRUNK/5382/]) HBASE-11333 Remove deprecated class MetaMigrationConvertingToPB (stack: rev 8372d9694d79e942570c3a89e531b96c13db1e08) * hbase-server/src/test/java/org/apache/hadoop/hbase/TestMetaMigrationConvertingToPB.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * hbase-server/src/test/data/TestMetaMigrationConvertToPB.README * hbase-server/src/main/java/org/apache/hadoop/hbase/MetaMigrationConvertingToPB.java HBASE-11333 Remove deprecated class MetaMigrationConvertingToPB (stack: rev 889c8a6f3a638e7eeffe5c0663ebd02e40a52c5a) * hbase-server/src/test/data/TestMetaMigrationConvertToPB.tgz > Remove deprecated class MetaMigrationConvertingToPB > --- > > Key: HBASE-11333 > URL: https://issues.apache.org/jira/browse/HBASE-11333 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov >Priority: Trivial > Fix For: 2.0.0 > > Attachments: 11333v2.txt, 11333v3.txt, HBASE-11333.patch > > > MetaMigrationConvertingToPB is marked deprecated and to be deleted next major > release after 0.96. Is that the time? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11697) Improve the 'Too many blocks' message on UI blockcache status page
[ https://issues.apache.org/jira/browse/HBASE-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090107#comment-14090107 ] Hudson commented on HBASE-11697: SUCCESS: Integrated in HBase-0.98 #441 (See [https://builds.apache.org/job/HBase-0.98/441/]) HBASE-11697 Improve the 'Too many blocks' message on UI blockcache status page (Mikhail Antonov) (apurtell: rev 6283ec730596b2e47666e8bef3a0fdb0442f55e6) * hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon > Improve the 'Too many blocks' message on UI blockcache status page > -- > > Key: HBASE-11697 > URL: https://issues.apache.org/jira/browse/HBASE-11697 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11697.patch, TooManyBlocks.png > > > If metrics calculations over blockcache contents stopped after examining > hbase.ui.blockcache.by.file.max items, the UI will put up a message. However, > this notion of "too many blocks" / fullness refers to structures used for > calculating blockcache metrics. See BlockCacheUtil. We should improve this > message so it does not leave a user the impression the blockcache may be in a > bad state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090106#comment-14090106 ] Hudson commented on HBASE-11678: SUCCESS: Integrated in HBase-0.98 #441 (See [https://builds.apache.org/job/HBase-0.98/441/]) HBASE-11678 BucketCache ramCache fills heap after running a few hours (stack: rev a41eca43e34dee7570e94a4f62080b4b541fbe7a) * hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketWriterThread.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestProtoBufRpc.java HBASE-11678 BucketCache ramCache fills heap after running a few hours -- 0.98 addendum (stack: rev b8e7c61645404d50fb247307ea590f71a6a7472e) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, 11678v3.098.txt, 11678v3.txt, 11678v3.txt, > gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11704) Include set-up instructions for building with IDEs besides Eclipse
Misty Stanley-Jones created HBASE-11704: --- Summary: Include set-up instructions for building with IDEs besides Eclipse Key: HBASE-11704 URL: https://issues.apache.org/jira/browse/HBASE-11704 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones We have pretty detailed instructions for setting up Eclipse to build HBase at http://hbase.apache.org/book/ides.html#eclipse. It would be nice to have this for other popular IDEs such as IntelliJ. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11697) Improve the 'Too many blocks' message on UI blockcache status page
[ https://issues.apache.org/jira/browse/HBASE-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090058#comment-14090058 ] Hudson commented on HBASE-11697: FAILURE: Integrated in HBase-1.0 #90 (See [https://builds.apache.org/job/HBase-1.0/90/]) HBASE-11697 Improve the 'Too many blocks' message on UI blockcache status page (Mikhail Antonov) (apurtell: rev 0c1daa6efdac8338c268da486de446f2c55d15ce) * hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/BlockCacheTmpl.jamon > Improve the 'Too many blocks' message on UI blockcache status page > -- > > Key: HBASE-11697 > URL: https://issues.apache.org/jira/browse/HBASE-11697 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11697.patch, TooManyBlocks.png > > > If metrics calculations over blockcache contents stopped after examining > hbase.ui.blockcache.by.file.max items, the UI will put up a message. However, > this notion of "too many blocks" / fullness refers to structures used for > calculating blockcache metrics. See BlockCacheUtil. We should improve this > message so it does not leave a user the impression the blockcache may be in a > bad state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090057#comment-14090057 ] Hudson commented on HBASE-11678: FAILURE: Integrated in HBase-1.0 #90 (See [https://builds.apache.org/job/HBase-1.0/90/]) HBASE-11678 BucketCache ramCache fills heap after running a few hours (stack: rev 2fea85a0b57c807e0ce57dac66493199ff52f0fc) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestProtoBufRpc.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java * hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestIPC.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketWriterThread.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestProcedureManager.java > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, 11678v3.098.txt, 11678v3.txt, 11678v3.txt, > gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-6626) Add a chapter on HDFS in the troubleshooting section of the HBase reference guide.
[ https://issues.apache.org/jira/browse/HBASE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090055#comment-14090055 ] Hudson commented on HBASE-6626: --- FAILURE: Integrated in HBase-1.0 #90 (See [https://builds.apache.org/job/HBase-1.0/90/]) HBASE-6626 Add a chapter on HDFS in the troubleshooting section of the HBase reference guide (Misty Stanley-Jones) (jmhsieh: rev 28e6aedf534ca3a7bfb7395544a4435fb763c772) * src/main/docbkx/troubleshooting.xml > Add a chapter on HDFS in the troubleshooting section of the HBase reference > guide. > -- > > Key: HBASE-6626 > URL: https://issues.apache.org/jira/browse/HBASE-6626 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.95.2 >Reporter: Nicolas Liochon >Assignee: Misty Stanley-Jones >Priority: Blocker > Fix For: 0.99.0, 2.0.0 > > Attachments: HBASE-6626.patch, troubleshooting.txt > > > I looked mainly at the major failure case, but here is what I have: > New sub chapter in the existing chapter "Troubleshooting and Debugging > HBase": "HDFS & HBASE" > 1) HDFS & HBase > 2) Connection related settings > 2.1) Number of retries > 2.2) Timeouts > 3) Log samples > 1) HDFS & HBase > HBase uses HDFS to store its HFile, i.e. the core HBase files and the > Write-Ahead-Logs, i.e. the files that will be used to restore the data after > a crash. > In both cases, the reliability of HBase comes from the fact that HDFS writes > the data to multiple locations. To be efficient, HBase needs the data to be > available locally, hence it's highly recommended to have the HDFS datanode on > the same machines as the HBase Region Servers. > Detailed information on how HDFS works can be found at [1]. > Important features are: > - HBase is a client application of HDFS, i.e. uses the HDFS DFSClient class. > This class can appears in HBase logs with other HDFS client related logs. > - Some HDFS settings are HDFS-server-side, i.e. must be set on the HDFS > side, while some other are HDFS-client-side, i.e. must be set in HBase, while > some other must be set in both places. > - the HDFS writes are pipelined from one datanode to another. When writing, > there are communications between: > - HBase and HDFS namenode, through the HDFS client classes. > - HBase and HDFS datanodes, through the HDFS client classes. > - HDFS datanode between themselves: issues on these communications are in > HDFS logs, not HBase. HDFS writes are always local when possible. As a > consequence, there should not be much write error in HBase Region Servers: > they write to the local datanode. If this datanode can't replicate the > blocks, it will appear in its logs, not in the region servers logs. > - datanodes can be contacted through the ipc.Client interface (once again > this class can shows up in HBase logs) and the data transfer interface > (usually shows up as the DataNode class in the HBase logs). There are on > different ports (defaults being: 50010 and 50020). > - To understand exactly what's going on, you must look that the HDFS log > files as well: HBase logs represent the client side. > - With the default setting, HDFS needs 630s to mark a datanode as dead. For > this reason, this node will still be tried by HBase or by other datanodes > when writing and reading until HDFS definitively decides it's dead. This will > add some extras lines in the logs. This monitoring is performed by the > NameNode. > - The HDFS clients (i.e. HBase using HDFS client code) don't fully rely on > the NameNode, but can mark temporally a node as dead if they had an error > when they tried to use it. > 2) Settings for retries and timeouts > 2.1) Retries > ipc.client.connect.max.retries > Default 10 > Indicates the number of retries a client will make to establish a server > connection. Not taken into account if the error is a SocketTimeout. In this > case the number of retries is 45 (fixed on branch, HADOOP-7932 or in > HADOOP-7397). For SASL, the number of retries is hard-coded to 15. Can be > increased, especially if the socket timeouts have been lowered. > ipc.client.connect.max.retries.on.timeouts > Default 45 > If you have HADOOP-7932, max number of retries on timeout. Counter is > different than ipc.client.connect.max.retries so if you mix the socket errors > you will get 55 retries with the default values. Could be lowered, once it is > available. With HADOOP-7397 ipc.client.connect.max.retries is reused so there > would be 10 tries. > dfs.client.block.write.retries > Default 3 > Number of tries for the client when writing a block. After a failure, will > connect to the namenode a get a new location, sending the list of the > datanodes already tried
[jira] [Created] (HBASE-11703) Meta region state could be corrupted
Jimmy Xiang created HBASE-11703: --- Summary: Meta region state could be corrupted Key: HBASE-11703 URL: https://issues.apache.org/jira/browse/HBASE-11703 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Internal meta region state could be corrupted if the meta is not on master: 1. the meta region server (not master) shuts down, 2. meta SSH offlines it without updating the dead server's region list, 3. meta is transitioned to pending_open and the previous server (the dead server) of meta is lost, 4. meta is assigned somewhere else without updating its previous server, 5. normal SSH processes the dead server and offlines all of it's dead regions including the meta, so the meta internal state is corrupted -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11701) Start and end of memstore flush log should be on the same level
[ https://issues.apache.org/jira/browse/HBASE-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11701: --- Fix Version/s: 0.98.6 2.0.0 0.99.0 +1 Will commit shortly unless objection. > Start and end of memstore flush log should be on the same level > --- > > Key: HBASE-11701 > URL: https://issues.apache.org/jira/browse/HBASE-11701 > Project: HBase > Issue Type: Improvement > Components: Usability >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11701.patch > > > Currently the start of the memstore flush is done in DEBUG level and when it > finishes is done in INFO. Both should be done in the same level. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11702) Better introspection of long running compactions
[ https://issues.apache.org/jira/browse/HBASE-11702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11702: --- Priority: Minor (was: Major) > Better introspection of long running compactions > > > Key: HBASE-11702 > URL: https://issues.apache.org/jira/browse/HBASE-11702 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > > For better introspection of long running compactions, periodically print > compaction progress for a file at DEBUG level (thread name, file path, total > compacted KVs, total compacted bytes, completion percent, rate) and expose > regionserver wide compaction metrics (KVs compacted and bytes compacted) as > Hadoop value metrics. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11702) Better introspection of long running compactions
Andrew Purtell created HBASE-11702: -- Summary: Better introspection of long running compactions Key: HBASE-11702 URL: https://issues.apache.org/jira/browse/HBASE-11702 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.99.0, 2.0.0, 0.98.6 For better introspection of long running compactions, periodically print compaction progress for a file at DEBUG level (thread name, file path, total compacted KVs, total compacted bytes, completion percent, rate) and expose regionserver wide compaction metrics (KVs compacted and bytes compacted) as Hadoop value metrics. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11700) Fix typo of hbase.store.compaction.ratio (should be hbase.hstore.compaction.ratio) in Ref Guide
[ https://issues.apache.org/jira/browse/HBASE-11700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11700: Labels: beginner (was: noob) > Fix typo of hbase.store.compaction.ratio (should be > hbase.hstore.compaction.ratio) in Ref Guide > --- > > Key: HBASE-11700 > URL: https://issues.apache.org/jira/browse/HBASE-11700 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Misty Stanley-Jones > Labels: beginner > > Hey, noticed that in the doc it lists out the following option: > hbase.store.compaction.ratio but it should be hbase.hstore.compaction.ratio. > Doc looks really good though! I will hopefully have more feedback for you > soon. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11699) Region servers exclusion list to HMaster.
[ https://issues.apache.org/jira/browse/HBASE-11699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089992#comment-14089992 ] Sean Busbey commented on HBASE-11699: - Keep in mind the impact of our [invariant on zookeeper state|http://hbase.apache.org/book.html#design.invariants.zk.data] being transient. Presuming we'd want to persist this information across zookeeper wipes, where could we persist this? Is the dataset small enough that a special file in HDFS works? Should this be stored in a special table? Can we add it to META? > Region servers exclusion list to HMaster. > - > > Key: HBASE-11699 > URL: https://issues.apache.org/jira/browse/HBASE-11699 > Project: HBase > Issue Type: New Feature > Components: Admin, Client, regionserver, Zookeeper >Affects Versions: 0.98.3 >Reporter: Gomathivinayagam Muthuvinayagam >Priority: Minor > Fix For: 0.98.3 > > Original Estimate: 96h > Remaining Estimate: 96h > > Currently HBase does not support adding set of region servers to be in the > exclusion list. So that administrators can prevent accidental startups of > some region servers to join the cluster. There was initially some work done, > and it is available in https://issues.apache.org/jira/browse/HBASE-3833. It > was not done after that. > I am planning to contribute it as a patch, and I would like to do some > improvements as well. Instead of storing the exclusion entries on a file, I > am planning to store it on zookeeper. Can anyone suggest thoughts on this? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11701) Start and end of memstore flush log should be on the same level
[ https://issues.apache.org/jira/browse/HBASE-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-11701: -- Attachment: HBASE-11701.patch > Start and end of memstore flush log should be on the same level > --- > > Key: HBASE-11701 > URL: https://issues.apache.org/jira/browse/HBASE-11701 > Project: HBase > Issue Type: Improvement > Components: Usability >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Attachments: HBASE-11701.patch > > > Currently the start of the memstore flush is done in DEBUG level and when it > finishes is done in INFO. Both should be done in the same level. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11701) Start and end of memstore flush log should be on the same level
[ https://issues.apache.org/jira/browse/HBASE-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-11701: -- Status: Patch Available (was: Open) > Start and end of memstore flush log should be on the same level > --- > > Key: HBASE-11701 > URL: https://issues.apache.org/jira/browse/HBASE-11701 > Project: HBase > Issue Type: Improvement > Components: Usability >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez >Priority: Trivial > Attachments: HBASE-11701.patch > > > Currently the start of the memstore flush is done in DEBUG level and when it > finishes is done in INFO. Both should be done in the same level. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11701) Start and end of memstore flush log should be on the same level
Esteban Gutierrez created HBASE-11701: - Summary: Start and end of memstore flush log should be on the same level Key: HBASE-11701 URL: https://issues.apache.org/jira/browse/HBASE-11701 Project: HBase Issue Type: Improvement Components: Usability Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Trivial Currently the start of the memstore flush is done in DEBUG level and when it finishes is done in INFO. Both should be done in the same level. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11695) PeriodicFlusher and WakeFrequency issues
[
https://issues.apache.org/jira/browse/HBASE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089955#comment-14089955
]
Lars Hofhansl commented on HBASE-11695:
---
Based on the observation that a 128mb memstore can be send in about 1s across a
1ge link, maybe the current jitter of 3-23s is good enough. 10-70s still seems
a safer bet, but anything between 5s and 5mins should be OK. :)
I still want adjust the wait-time to be greater than the expected jitter in
order to avoid confusion for folks in the future.
> PeriodicFlusher and WakeFrequency issues
>
>
> Key: HBASE-11695
> URL: https://issues.apache.org/jira/browse/HBASE-11695
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.21
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.99.0, 2.0.0, 0.94.23, 0.98.6
>
> Attachments: 11695-trunk.txt
>
>
> We just ran into a flush storm caused by the PeriodicFlusher.
> Many memstore became eligible for flushing at exactly the same time, the
> effect we've seen is that the exact same region was flushed multiple times,
> because the flusher wakes up too often (every 10s). The jitter of 20s is
> larger than that and it takes some time to actually flush the memstore.
> Here's one example. We've seen 100's of these, monopolizing the flush queue
> and preventing "important" flushes from happening.
> {code}
> 06-Aug-2014 20:11:56 [regionserver60020.periodicFlusher] INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer[1397]--
> regionserver60020.periodicFlusher requesting flush for region
> tsdb,\x00\x00\x0AO\xCF*
> \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2.
> after a delay of 13449
> 06-Aug-2014 20:12:06 [regionserver60020.periodicFlusher] INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer[1397]--
> regionserver60020.periodicFlusher requesting flush for region
> tsdb,\x00\x00\x0AO\xCF*
> \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2.
> after a delay of 14060
> {code}
> So we need to increase the period of the PeriodicFlusher to at least the
> random jitter, also increase the default random jitter (20s does not help
> with many regions).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11691) Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089953#comment-14089953 ] Nick Dimiduk commented on HBASE-11691: -- It shouldn't impact, but PE is pretty fragile. This patch tweaks the job creation code, which is why I pointed it out. You're probably right though. Anyway +1 stands. > Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance > Evaluation) to 0.94 > --- > > Key: HBASE-11691 > URL: https://issues.apache.org/jira/browse/HBASE-11691 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11691.patch > > > Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance > Evaluation) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11690) Backport HBASE-5934 (Add the ability for Performance Evaluation to set the table compression) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-11690. Resolution: Fixed Fix Version/s: 0.94.23 Hadoop Flags: Reviewed Pushed to 0.94. That import is needed after all. > Backport HBASE-5934 (Add the ability for Performance Evaluation to set the > table compression) to 0.94 > - > > Key: HBASE-11690 > URL: https://issues.apache.org/jira/browse/HBASE-11690 > Project: HBase > Issue Type: Task >Affects Versions: 0.94.23 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11690.patch > > > Backport HBASE-5934 (Add the ability for Performance Evaluation to set the > table compression) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11691) Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance Evaluation) to 0.94
[ https://issues.apache.org/jira/browse/HBASE-11691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-11691. Resolution: Fixed Hadoop Flags: Reviewed Pushed to 0.94 > Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance > Evaluation) to 0.94 > --- > > Key: HBASE-11691 > URL: https://issues.apache.org/jira/browse/HBASE-11691 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.94.23 > > Attachments: HBASE-11691.patch > > > Backport HBASE-7156 (Add Data Block Encoding and -D opts to Performance > Evaluation) to 0.94 branch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11661) Quickstart chapter claims standalone mode has multiple processes
[
https://issues.apache.org/jira/browse/HBASE-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089941#comment-14089941
]
Hudson commented on HBASE-11661:
SUCCESS: Integrated in HBase-TRUNK #5381 (See
[https://builds.apache.org/job/HBase-TRUNK/5381/])
HBASE-11661 standalone verification should only look for a single process.
(stack: rev b2cc061460db0c204b30cee4a12958838a171905)
* src/main/docbkx/getting_started.xml
> Quickstart chapter claims standalone mode has multiple processes
>
>
> Key: HBASE-11661
> URL: https://issues.apache.org/jira/browse/HBASE-11661
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE_11661-v1.patch
>
>
> The quickstart chapter on launching a standalone hbase says to validate the
> launch by checking on both the HMaster and HRegionServer processes:
> {quote}
> 4. The bin/start-hbase.sh script is provided as a convenient way to start
> HBase. Issue the command, and if all goes well, a message is logged to
> standard output showing that HBase started successfully. You can use the jps
> command to verify that you have one running process called HMaster and at
> least one called HRegionServer.
> {quote}
> This is incorrect. In standalone mode there is only a single HMaster process.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-11697) Improve the 'Too many blocks' message on UI blockcache status page
[ https://issues.apache.org/jira/browse/HBASE-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-11697. Resolution: Fixed Hadoop Flags: Reviewed Committed to 0.98+ > Improve the 'Too many blocks' message on UI blockcache status page > -- > > Key: HBASE-11697 > URL: https://issues.apache.org/jira/browse/HBASE-11697 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11697.patch, TooManyBlocks.png > > > If metrics calculations over blockcache contents stopped after examining > hbase.ui.blockcache.by.file.max items, the UI will put up a message. However, > this notion of "too many blocks" / fullness refers to structures used for > calculating blockcache metrics. See BlockCacheUtil. We should improve this > message so it does not leave a user the impression the blockcache may be in a > bad state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11697) Improve the 'Too many blocks' message on UI blockcache status page
[ https://issues.apache.org/jira/browse/HBASE-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089929#comment-14089929 ] Andrew Purtell commented on HBASE-11697: Agree, I don't think we need the message, but this change has 3 +1s so I'm committing it now. > Improve the 'Too many blocks' message on UI blockcache status page > -- > > Key: HBASE-11697 > URL: https://issues.apache.org/jira/browse/HBASE-11697 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11697.patch, TooManyBlocks.png > > > If metrics calculations over blockcache contents stopped after examining > hbase.ui.blockcache.by.file.max items, the UI will put up a message. However, > this notion of "too many blocks" / fullness refers to structures used for > calculating blockcache metrics. See BlockCacheUtil. We should improve this > message so it does not leave a user the impression the blockcache may be in a > bad state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11700) Fix typo of hbase.store.compaction.ratio (should be hbase.hstore.compaction.ratio) in Ref Guide
Misty Stanley-Jones created HBASE-11700: --- Summary: Fix typo of hbase.store.compaction.ratio (should be hbase.hstore.compaction.ratio) in Ref Guide Key: HBASE-11700 URL: https://issues.apache.org/jira/browse/HBASE-11700 Project: HBase Issue Type: Bug Components: documentation Reporter: Misty Stanley-Jones Hey, noticed that in the doc it lists out the following option: hbase.store.compaction.ratio but it should be hbase.hstore.compaction.ratio. Doc looks really good though! I will hopefully have more feedback for you soon. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11695) PeriodicFlusher and WakeFrequency issues
[
https://issues.apache.org/jira/browse/HBASE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-11695:
--
Priority: Minor (was: Critical)
> PeriodicFlusher and WakeFrequency issues
>
>
> Key: HBASE-11695
> URL: https://issues.apache.org/jira/browse/HBASE-11695
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.21
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Minor
> Fix For: 0.99.0, 2.0.0, 0.94.23, 0.98.6
>
> Attachments: 11695-trunk.txt
>
>
> We just ran into a flush storm caused by the PeriodicFlusher.
> Many memstore became eligible for flushing at exactly the same time, the
> effect we've seen is that the exact same region was flushed multiple times,
> because the flusher wakes up too often (every 10s). The jitter of 20s is
> larger than that and it takes some time to actually flush the memstore.
> Here's one example. We've seen 100's of these, monopolizing the flush queue
> and preventing "important" flushes from happening.
> {code}
> 06-Aug-2014 20:11:56 [regionserver60020.periodicFlusher] INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer[1397]--
> regionserver60020.periodicFlusher requesting flush for region
> tsdb,\x00\x00\x0AO\xCF*
> \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2.
> after a delay of 13449
> 06-Aug-2014 20:12:06 [regionserver60020.periodicFlusher] INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer[1397]--
> regionserver60020.periodicFlusher requesting flush for region
> tsdb,\x00\x00\x0AO\xCF*
> \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2.
> after a delay of 14060
> {code}
> So we need to increase the period of the PeriodicFlusher to at least the
> random jitter, also increase the default random jitter (20s does not help
> with many regions).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11695) PeriodicFlusher and WakeFrequency issues
[
https://issues.apache.org/jira/browse/HBASE-11695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089923#comment-14089923
]
Lars Hofhansl commented on HBASE-11695:
---
Can't try on the production site easily right now.
This was a dud anyway, in the sense that it does not cause the issue. It's just
a weirdness observed. The problem was that flushes took a very long time
(hours); not sure why, yet, but probably due to a networking issue. Hence all
flushes were waiting and after one hour all the waiting regions become eligible
for the periodic flusher.
The problem here is only a cosmetic problem then. Because the wake waittime is
less than the jitter in most of the cases we'll see each region requesting a
flush twice.
> PeriodicFlusher and WakeFrequency issues
>
>
> Key: HBASE-11695
> URL: https://issues.apache.org/jira/browse/HBASE-11695
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.21
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
>Priority: Critical
> Fix For: 0.99.0, 2.0.0, 0.94.23, 0.98.6
>
> Attachments: 11695-trunk.txt
>
>
> We just ran into a flush storm caused by the PeriodicFlusher.
> Many memstore became eligible for flushing at exactly the same time, the
> effect we've seen is that the exact same region was flushed multiple times,
> because the flusher wakes up too often (every 10s). The jitter of 20s is
> larger than that and it takes some time to actually flush the memstore.
> Here's one example. We've seen 100's of these, monopolizing the flush queue
> and preventing "important" flushes from happening.
> {code}
> 06-Aug-2014 20:11:56 [regionserver60020.periodicFlusher] INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer[1397]--
> regionserver60020.periodicFlusher requesting flush for region
> tsdb,\x00\x00\x0AO\xCF*
> \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2.
> after a delay of 13449
> 06-Aug-2014 20:12:06 [regionserver60020.periodicFlusher] INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer[1397]--
> regionserver60020.periodicFlusher requesting flush for region
> tsdb,\x00\x00\x0AO\xCF*
> \x00\x00\x01\x00\x01\x1F\x00\x00\x03\x00\x00\x0C,1340147003629.ef4a680b962592de910d0fdeb376dfc2.
> after a delay of 14060
> {code}
> So we need to increase the period of the PeriodicFlusher to at least the
> random jitter, also increase the default random jitter (20s does not help
> with many regions).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11697) Improve the 'Too many blocks' message on UI blockcache status page
[ https://issues.apache.org/jira/browse/HBASE-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089916#comment-14089916 ] Lars Hofhansl commented on HBASE-11697: --- +1 (Do we need that message at all? We're just getting some stats, doesn't matter than we only sampled a subset.) > Improve the 'Too many blocks' message on UI blockcache status page > -- > > Key: HBASE-11697 > URL: https://issues.apache.org/jira/browse/HBASE-11697 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11697.patch, TooManyBlocks.png > > > If metrics calculations over blockcache contents stopped after examining > hbase.ui.blockcache.by.file.max items, the UI will put up a message. However, > this notion of "too many blocks" / fullness refers to structures used for > calculating blockcache metrics. See BlockCacheUtil. We should improve this > message so it does not leave a user the impression the blockcache may be in a > bad state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Attachment: 11678v3.098.txt Commit to 0.98 needed an addendum. Here is what I applied. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, 11678v3.098.txt, 11678v3.txt, 11678v3.txt, > gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11678) BucketCache ramCache fills heap after running a few hours
[ https://issues.apache.org/jira/browse/HBASE-11678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11678: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to 0.98+. Cherry-picked. Fixed javadoc warning on commit. > BucketCache ramCache fills heap after running a few hours > - > > Key: HBASE-11678 > URL: https://issues.apache.org/jira/browse/HBASE-11678 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 0.99.0, 0.98.5, 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: > 0001-When-we-failed-add-an-entry-failing-with-a-CacheFull.patch, 11678v2.txt, > 11678v2.txt, 11678v3.txt, 11678v3.txt, > gc_crash_unevenblocks_with_lots_of_evictions.png, > gc_over_12_hours_unevenblocks_with_lots_of_evictions.png > > > Testing BucketCache, my heap filled after running for hours. Dumping heap, > culprit is the ramCache Map in BucketCache. Tried running with more writer > threads but made no difference. Trying to figure now how our accounting is > going wonky. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11697) Improve the 'Too many blocks' message on UI blockcache status page
[ https://issues.apache.org/jira/browse/HBASE-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089882#comment-14089882 ] stack commented on HBASE-11697: --- +1 Thanks for cleaning up my scary message [~mantonov] > Improve the 'Too many blocks' message on UI blockcache status page > -- > > Key: HBASE-11697 > URL: https://issues.apache.org/jira/browse/HBASE-11697 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Mikhail Antonov >Priority: Minor > Fix For: 0.99.0, 2.0.0, 0.98.6 > > Attachments: HBASE-11697.patch, TooManyBlocks.png > > > If metrics calculations over blockcache contents stopped after examining > hbase.ui.blockcache.by.file.max items, the UI will put up a message. However, > this notion of "too many blocks" / fullness refers to structures used for > calculating blockcache metrics. See BlockCacheUtil. We should improve this > message so it does not leave a user the impression the blockcache may be in a > bad state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11699) Region servers exclusion list to HMaster.
Gomathivinayagam Muthuvinayagam created HBASE-11699: --- Summary: Region servers exclusion list to HMaster. Key: HBASE-11699 URL: https://issues.apache.org/jira/browse/HBASE-11699 Project: HBase Issue Type: New Feature Components: Admin, Client, regionserver, Zookeeper Affects Versions: 0.98.3 Reporter: Gomathivinayagam Muthuvinayagam Priority: Minor Fix For: 0.98.3 Currently HBase does not support adding set of region servers to be in the exclusion list. So that administrators can prevent accidental startups of some region servers to join the cluster. There was initially some work done, and it is available in https://issues.apache.org/jira/browse/HBASE-3833. It was not done after that. I am planning to contribute it as a patch, and I would like to do some improvements as well. Instead of storing the exclusion entries on a file, I am planning to store it on zookeeper. Can anyone suggest thoughts on this? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-9005) Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete markers
[ https://issues.apache.org/jira/browse/HBASE-9005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089869#comment-14089869 ] Jonathan Hsieh commented on HBASE-9005: --- can you add a link to http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/HColumnDescriptor.html#KEEP_DELETED_CELLS instead of the current generic link to HCD (which has no javadoc)? Pointing to it and saying it is a boolean makes it much more actionable. Bonus points if an example were provided to change it in the shell. (maybe not just here, but > Improve documentation around KEEP_DELETED_CELLS, time range scans, and delete > markers > - > > Key: HBASE-9005 > URL: https://issues.apache.org/jira/browse/HBASE-9005 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Lars Hofhansl >Assignee: Jonathan Hsieh >Priority: Minor > Fix For: 0.99.0 > > Attachments: 9005.txt, HBASE-9005-1.patch > > > Without KEEP_DELETED_CELLS all timerange queries are broken if their range > covers a delete marker. > As some internal discussions with colleagues showed, this feature is not well > understand and documented. -- This message was sent by Atlassian JIRA (v6.2#6252)
